mlsanityData to 14 Oct 2026

VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding

arXiv 2412.03735 ↗First posted 4 Dec 2024cs.CVcited by 68

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal large language models (MLLMs) have recently shown significant advancements in video understanding, excelling in content reasoning and instruction-following tasks. However, hallucination, where models generate inaccurate or misleading content, remains underexplored in the video domain. Building on the observation that MLLM visual encoders often fail to distinguish visually different yet semantically similar video pairs, we introduce VidHalluc, the largest benchmark designed to examine hallucinations in MLLMs for video understanding. It consists of 5,002 videos, paired to highlight cases prone to hallucinations. VidHalluc assesses hallucinations across three critical dimensions: (1) action, (2) temporal sequence, and (3) scene transition. Comprehensive testing shows that most MLLMs are vulnerable to hallucinations across these dimensions. Furthermore, we propose DINO-HEAL, a training-free method that reduces hallucinations by incorporating spatial saliency from DINOv2 to reweight visual features during inference. Our results show that DINO-HEAL consistently improves performance on VidHalluc, achieving an average improvement of 3.02% in mitigating hallucinations across all tasks. Both the VidHalluc benchmark and DINO-HEAL code are available at https://people-robots.github.io/vidhalluc.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 79 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

70 of 79 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMs25 Sep
  2. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models24 Jun
  3. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  4. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  7. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  8. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  9. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  13. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  14. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  15. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  17. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  18. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  19. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  20. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  21. 2023Instruction Tuning with GPT-4cited by 686
  22. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  23. 2018Object Hallucination in Image Captioningcited by 669
  24. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  25. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  26. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  27. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  28. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  29. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  30. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  31. 2023Video Understanding with Large Language Models: A Surveycited by 206
  32. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  33. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  34. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  35. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  36. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  37. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  38. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 46.

68 papers.

Sort:

By area:

  1. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 18
  2. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 17
  3. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  4. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  5. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understandingcited by 10
  6. 2025SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decodingcited by 10
  7. 2025SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapsecited by 9
  8. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  9. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  10. 2025Self-alignment of Large Video Language Models with Refined Regularized Preference Optimizationcited by 7
  11. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  12. 2026Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflictscited by 7
38 more
  1. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoningcited by 5
  2. 2025SmokeBench: Evaluating Multimodal Large Language Models for Wildfire Smoke Detectioncited by 5
  3. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  4. 2026CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Modelscited by 4
  5. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  7. 2025Harnessing Meta-Learning for Controllable Full-Frame Video Stabilizationcited by 2
  8. 2025MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Modelscited by 2
  9. 2025NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Modelscited by 2
  10. 2026VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understandingcited by 2
  11. 2026Reinforcing Consistency in Video MLLMs with Structured Rewardscited by 2
  12. 2026Hilbert-Geo: Solving Solid Geometric Problems by Neural-Symbolic Reasoningcited by 2
  13. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 2
  14. 2025FIFA: Unified Faithfulness Evaluation Framework for Text-to-Video and Video-to-Text Generationcited by 1
  15. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  16. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Modelscited by 1
  17. 2026Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Modelscited by 1
  18. 2026Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Modelscited by 1
  19. 2026Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raterscited by 1
  20. 2026VideoSTF: Stress-Testing Output Repetition in Video Large Language Modelscited by 1
  21. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  22. 2026STEAR: Layer-Aware Spatiotemporal Evidence Intervention for Hallucination Mitigation in Video Large Language Modelscited by 1
  23. 2026Distorted or Fabricated? A Survey on Hallucination in Video LLMscited by 1
  24. 2026Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Modelscited by 1
  25. 2026CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMscited by 1
  26. 2026Towards Temporal Compositional Reasoning in Long-Form Sports Videoscited by 1
  27. 2026CRAFT: Critic-Refined Adaptive Key-Frame Targeting for Multimodal Video Question Answeringcited by 1
  28. 2026Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learningcited by 1
  29. 2026MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Modelscited by 1
  30. 2026Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Modelscited by 1
  31. 2025ArxEval: Evaluating Retrieval and Generation in Language Models for Scientific Literaturecited by 0
  32. 2025Decoupling Perception from Reasoning for Hallucination-Resistant Video Understandingcited by 0
  33. 2026VERHallu: Evaluating and Mitigating Event Relation Hallucination in Video Large Language Modelscited by 0
  34. 2026Logi-PAR: Logic-Infused Patient Activity Recognition via Differentiable Rulecited by 0
  35. 2026AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecastingcited by 0
  36. 2026Revealing Multi-View Hallucination in Large Vision-Language Modelscited by 0
  37. 2026Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Modelscited by 0
  38. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation
  2. 2025ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations
  3. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  4. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images
  5. 2024Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  7. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
  8. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Models