mlsanityData to 7 Oct 2026

Multi-Object Hallucination in Vision-Language Models

arXiv 2407.06192 ↗First posted 8 Jul 2024cs.CV cs.AI cs.CLcited by 60

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision language models (LVLMs) often suffer from object hallucination, producing objects not present in the given images. While current benchmarks for object hallucination primarily concentrate on the presence of a single object class rather than individual entities, this work systematically investigates multi-object hallucination, examining how models misperceive (e.g., invent nonexistent objects or become distracted) when tasked with focusing on multiple objects simultaneously. We introduce Recognition-based Object Probing Evaluation (ROPE), an automated evaluation protocol that considers the distribution of object classes within a single image during testing and uses visual referring prompts to eliminate ambiguity. With comprehensive empirical studies and analysis of potential factors leading to multi-object hallucination, we found that (1). LVLMs suffer more hallucinations when focusing on multiple objects compared to a single object. (2). The tested object class distribution affects hallucination behaviors, indicating that LVLMs may follow shortcuts and spurious correlations. (3). Hallucinatory behaviors are influenced by data-specific factors, salience and frequency, and model intrinsic behaviors. We hope to enable LVLMs to recognize and reason about multiple objects that often occur in realistic visual scenes, provide insights, and quantify our progress towards mitigating the issues.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 64 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

42 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  4. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  6. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
24 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  5. 2018Object Hallucination in Image Captioningcited by 661
  6. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  11. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  12. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  13. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  14. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  15. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  16. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  17. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  18. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  19. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  20. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
  21. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  22. 2023ImageNet-Hard: The Hardest Images Remaining from a Study of the Power of Zoom and Spatial Biases in Image Classificationcited by 33
  23. 2021Mitigating the Bias of Centered Objects in Common Datasetscited by 7
  24. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 30; 2026: 24.

60 papers.

Sort:

By area:

  1. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  2. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  3. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucinationcited by 33
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  5. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  6. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  7. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  8. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  9. 2025Using Vision Language Models for Safety Hazard Identification in Constructioncited by 12
  10. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  11. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  12. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
38 more
  1. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  2. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Modelscited by 7
  3. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  4. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  5. 2025CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methodscited by 5
  6. 2025The Mechanistic Emergence of Symbol Grounding in Language Modelscited by 5
  7. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Studycited by 4
  8. 2026FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMscited by 4
  9. 2025EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgerycited by 3
  10. 2025Attention Guided Alignment in Efficient Vision-Language Modelscited by 3
  11. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 3
  12. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinationscited by 3
  13. 2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Modelscited by 2
  14. 2025RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregationcited by 2
  15. 2025ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Modelscited by 2
  16. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  17. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  18. 2026Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMscited by 2
  19. 2026MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialoguecited by 2
  20. 2026SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineeringcited by 2
  21. 2024Intriguing Properties of Large Language and Vision Modelscited by 1
  22. 2025What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models?cited by 1
  23. 2025Prior Distribution and Model Confidencecited by 1
  24. 2025Detecting and Mitigating Insertion Hallucination in Video-to-Audio Generationcited by 1
  25. 2025Robust Driving QA through Metadata-Grounded Context and Task-Specific Promptscited by 1
  26. 2025What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Promptcited by 1
  27. 2025MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoningcited by 1
  28. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 1
  29. 2026Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Modelscited by 1
  30. 2026Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectificationcited by 1
  31. 2026Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictionscited by 1
  32. 2026Breaking the Illusion: When Positive Meets Negative in Multimodal Decodingcited by 1
  33. 2026Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learningcited by 1
  34. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 1
  35. 2025PGDA-KGQA: A Prompt-Guided Generative Framework with Multiple Data Augmentation Strategies for Knowledge Graph Question Answeringcited by 0
  36. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  37. 2025Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflectioncited by 0
  38. 2025A Video Is Not Worth a Thousand Wordscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating Object Hallucination in Large Vision-Language Models
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
  3. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  4. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  6. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  7. 2025Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
  8. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective