mlsanityData to 7 Oct 2026

VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models

arXiv 2406.16338 ↗First posted 24 Jun 2024cs.CVcited by 88

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have extended their capabilities to video understanding. Yet, these models are often plagued by "hallucinations", where irrelevant or nonsensical content is generated, deviating from the actual video context. This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs). VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis, including object-relation, temporal, semantic detail, extrinsic factual, and extrinsic non-factual hallucinations. We adopt an adversarial binary VideoQA method for comprehensive evaluation, where pairs of basic and hallucinated questions are crafted strategically. By evaluating eleven LVLMs on VideoHallucer, we reveal that i) the majority of current models exhibit significant issues with hallucinations; ii) while scaling datasets and parameters improves models' ability to detect basic visual cues and counterfactuals, it provides limited benefit for detecting extrinsic factual hallucinations; iii) existing models are more adept at detecting facts than identifying hallucinations. As a byproduct, these analyses further instruct the development of our self-PEP framework, achieving an average of 5.38% improvement in hallucination resistance across all model architectures.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 75 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

57 of 75 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset17 Mar
  2. 2024Visual Hallucinations of Multi-modal Large Language Models22 Feb
  3. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  4. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  5. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  6. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges6 Nov
  7. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  9. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  10. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 47

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
35 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  5. 2017Dense-Captioning Events in Videoscited by 789
  6. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  11. 2018Object Hallucination in Image Captioningcited by 661
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  13. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  14. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  15. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  16. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  17. 2023How Language Model Hallucinations Can Snowballcited by 310
  18. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  19. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  20. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  21. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  22. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  23. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  24. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  25. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  26. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  27. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  28. 2020Controlled Hallucinations: Learning to Generate Faithfully from Noisy Datacited by 83
  29. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  30. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  31. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  32. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 44
  33. 2020A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systemscited by 34
  34. 2023Learning to Simulate Natural Language Feedback for Interactive Semantic Parsingcited by 19
  35. 2020Classification of Important Segments in Educational Videos using Multimodal Featurescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 35; 2026: 38.

88 papers.

Sort:

By area:

  1. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  2. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  3. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  6. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  7. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  8. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  9. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  10. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  11. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  12. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
38 more
  1. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  4. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  5. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  6. 2024Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Modelscited by 18
  7. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  8. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  9. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  10. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  11. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  12. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  13. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  14. 2025Video SimpleQA: Towards Factuality Evaluation in Large Video Language Modelscited by 11
  15. 2024DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understandingcited by 10
  16. 2024Progress-Aware Video Frame Captioningcited by 10
  17. 2025Tackling Hallucination from Conditional Models for Medical Image Reconstruction with DynamicDPScited by 10
  18. 2025EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understandingcited by 10
  19. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understandingcited by 10
  20. 2025SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decodingcited by 8
  21. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 7
  22. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  23. 2025Self-alignment of Large Video Language Models with Refined Regularized Preference Optimizationcited by 6
  24. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  25. 2025EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Modelscited by 5
  26. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  27. 2025VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Modelscited by 4
  28. 2026CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Modelscited by 4
  29. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understandingcited by 4
  30. 2024ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Modelscited by 3
  31. 2024Understanding Multimodal Hallucination with Parameter-Free Representation Alignmentcited by 2
  32. 2025TIME: Temporal-Sensitive Multi-Dimensional Instruction Tuning and Robust Benchmarking for Video-LLMscited by 2
  33. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  34. 2025NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Modelscited by 2
  35. 2025Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignmentcited by 2
  36. 2026Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularizationcited by 2
  37. 2026VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understandingcited by 2
  38. 2026Reinforcing Consistency in Video MLLMs with Structured Rewardscited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations
  2. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models
  3. 2025A Survey of Multimodal Hallucination Evaluation and Detection
  4. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediations
  5. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  6. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
  7. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  8. 2024A Survey on Hallucination in Large Vision-Language Models