mlsanityData to 14 Oct 2026

Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis

arXiv 2412.02946 ↗First posted 4 Dec 2024cs.CV cs.AI cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Recent advancements in large vision-language models (LVLM) have significantly enhanced their ability to comprehend visual inputs alongside natural language. However, a major challenge in their real-world application is hallucination, where LVLMs generate non-existent visual elements, eroding user trust. The underlying mechanism driving this multimodal hallucination is poorly understood. Minimal research has illuminated whether contexts such as sky, tree, or grass field involve the LVLM in hallucinating a frisbee. We hypothesize that hidden factors, such as objects, contexts, and semantic foreground-background structures, induce hallucination. This study proposes a novel causal approach: a hallucination probing system to identify these hidden factors. By analyzing the causality between images, text prompts, and network saliency, we systematically explore interventions to block these factors. Our experimental findings show that a straightforward technique based on our analysis can significantly reduce hallucinations. Additionally, our analyses indicate the potential to edit network internals to minimize hallucinated outputs.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 54 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

46 of 54 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  2. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models22 Apr
  3. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  11. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  12. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
5 more
  1. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  2. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  3. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  4. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2018Object Hallucination in Image Captioningcited by 669
17 more
  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  2. 2023Function Vectors in Large Language Modelscited by 308
  3. 2013Direct and Indirect Effectscited by 280
  4. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  5. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  6. 2021Causal Attention for Vision-Language Taskscited by 111
  7. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  8. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  9. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  10. 2020Deconfounded Image Captioning: A Causal Retrospectcited by 65
  11. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  12. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  13. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  14. 2023Can We Edit Multimodal Large Language Models?cited by 44
  15. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  16. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modelingcited by 34
  17. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  2. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 4
  3. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  4. 2025How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucination
  2. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
  3. 2024Multi-Object Hallucination in Vision-Language Models
  4. 2025Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMs
  5. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  6. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  7. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  8. 2025Causal-HalBench: Uncovering LVLMs Object Hallucinations Through Causal Intervention