# VidHal: Benchmarking Temporal Hallucinations in Vision LLMs

arXiv: https://arxiv.org/abs/2411.16771 · HTML: https://mlsanity.com/paper/2411.16771/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 88 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Builds on (most cited first, 30 of 73)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)

## Built on by (most cited first, 19 of 19)

- [Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark](https://mlsanity.com/paper/2504.14693/) (2025)
- [Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning](https://mlsanity.com/paper/2506.00318/) (2025)
- [V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models](https://mlsanity.com/paper/2511.16668/) (2025)
- [ARGUS: Hallucination and Omission Evaluation in Video-LLMs](https://mlsanity.com/paper/2506.07371/) (2025)
- [Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Grounding](https://mlsanity.com/paper/2509.11866/) (2025)
- [VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment](https://mlsanity.com/paper/2504.14096/) (2025)
- [Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering](https://mlsanity.com/paper/2505.12826/) (2025)
- [XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models](https://mlsanity.com/paper/2510.15148/) (2025)
- [Mitigating Object and Action Hallucinations in Multimodal LLMs via Self-Augmented Contrastive Alignment](https://mlsanity.com/paper/2512.04356/) (2025)
- [MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models](https://mlsanity.com/paper/2509.08538/) (2025)
- [NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Models](https://mlsanity.com/paper/2511.06475/) (2025)
- [No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs](https://mlsanity.com/paper/2606.31933/) (2026)
- [Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models](https://mlsanity.com/paper/2602.00559/) (2026)
- [Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models](https://mlsanity.com/paper/2602.11244/) (2026)
- [Distorted or Fabricated? A Survey on Hallucination in Video LLMs](https://mlsanity.com/paper/2604.12944/) (2026)
- [CCTVBench: Contrastive Consistency Traffic VideoQA Benchmark for Multimodal LLMs](https://mlsanity.com/paper/2604.20460/) (2026)
- [Video Evidence to Reasoning Efficient Video Understanding via Explicit Evidence Grounding](https://mlsanity.com/paper/2601.07761/) (2026)
- [Revealing Multi-View Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2603.23934/) (2026)
- [CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning](https://mlsanity.com/paper/2605.26967/) (2026)

## Abstract (arXiv)

Vision Large Language Models (VLLMs) are widely acknowledged to be prone to hallucinations. Existing research addressing this problem has primarily been confined to image inputs, with limited exploration of video-based hallucinations. Furthermore, current evaluation methods fail to capture nuanced errors in generated responses, which are often exacerbated by the rich spatiotemporal dynamics of videos. To address this, we introduce VidHal, a benchmark specially designed to evaluate video-based hallucinations in VLLMs. VidHal is constructed by bootstrapping video instances across a wide range of common temporal aspects. A defining feature of our benchmark lies in the careful creation of captions which represent varying levels of hallucination associated with each video. To enable fine-grained evaluation, we propose a novel caption ordering task requiring VLLMs to rank captions by hallucinatory extent. We conduct extensive experiments on VidHal and comprehensively evaluate a broad selection of models. Our results uncover significant limitations in existing VLLMs regarding hallucination generation. Through our benchmark, we aim to inspire further research on 1) holistic understanding of VLLM capabilities, particularly regarding hallucination, and 2) extensive development of advanced VLLMs to alleviate this problem.