mlsanityData to 7 Oct 2026

HallE-Control: Controlling Object Hallucination in Large Multimodal Models

arXiv 2310.01779 ↗First posted 3 Oct 2023cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning. To address this, we introduce textitCCEval, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we make the first attempt to investigate such hallucination from different aspects, including image resolution, the language decoder size, and instruction data amount, quality, granularity. Our findings underscore the unwarranted inference when the language description includes details at a finer object granularity than what the vision module can ground or verify, thus inducing hallucination. To control such hallucinations, we further attribute the reliability of captioning to contextual knowledge (involving only contextually grounded objects) and parametric knowledge (containing inferred objects by the model). Thus, we introduce textitHallE-Control, a controllable LMM in terms of Hallucination in object Existence. HallE-Control can condition the captioning to shift between (i) exclusively depicting contextual knowledge for grounded objects and (ii) blending it with parametric knowledge to imagine inferred objects. Our method reduces hallucination by 44% compared to LLaVA7B and maintains the object coverage.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 61 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

50 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  2. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 48

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
36 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  15. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  16. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  17. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  18. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  19. 2018Object Hallucination in Image Captioningcited by 661
  20. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  21. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  22. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  23. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  24. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  25. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  26. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  27. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  28. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  29. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  30. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  31. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  32. 2023Word Embeddings Are Steers for Language Modelscited by 70
  33. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  34. 2021Controlling Hallucinations at Word Level in Data-to-Text Generationcited by 41
  35. 2022Neural Pipeline for Zero-Shot Data-to-Text Generationcited by 19
  36. 2023AVIS: Autonomous Visual Information Seeking with Large Language Model Agentcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024An Introduction to Vision-Language Modelingcited by 175
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  3. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  4. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  5. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  6. 2024HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoningcited by 30
  7. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  8. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 14
  9. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understandingcited by 11
  10. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  11. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  12. 2025LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generationcited by 8
19 more
  1. 2025Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucinationcited by 7
  2. 2025ZINA: Multimodal Fine-grained Hallucination Detection and Editingcited by 5
  3. 2024Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Languagecited by 4
  4. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Modelscited by 4
  5. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  6. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  7. 2025Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Modelscited by 2
  8. 2025Stop learning it all to mitigate visual hallucination, Focus on the hallucination targetcited by 2
  9. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  10. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflectioncited by 2
  11. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 2
  12. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  13. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  14. 2026RSHallu: Dual-Mode Hallucination Evaluation for Remote-Sensing Multimodal Large Language Models with Domain-Tailored Mitigationcited by 1
  15. 2026Unified Hallucination Fuzzing for Multimodal Large Language Modelscited by 1
  16. 2026Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Modelscited by 0
  17. 2026Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruptioncited by 0
  18. 2026Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generationcited by 0
  19. 2026Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splittingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  2. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
  3. 2024Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?
  4. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
  5. 2026Fine-Grained Multi Image Object Hallucination Benchmark
  6. 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
  7. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens