# Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?

arXiv: https://arxiv.org/abs/2505.14321 · HTML: https://mlsanity.com/paper/2505.14321/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution](https://mlsanity.com/paper/2409.12961/) (2024)
- [SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models](https://mlsanity.com/paper/2407.15841/) (2024)
- [Apollo: An Exploration of Video Understanding in Large Multimodal Models](https://mlsanity.com/paper/2412.10360/) (2024)
- [LIME: Less Is More for MLLM Evaluation](https://mlsanity.com/paper/2409.06851/) (2024)
- [Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models](https://mlsanity.com/paper/2406.08024/) (2024)

## Built on by (most cited first, 20 of 20)

- [HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering](https://mlsanity.com/paper/2512.14870/) (2025)
- [NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding](https://mlsanity.com/paper/2601.01095/) (2026)
- [How Do Video Foundation Models Encode Intuitive Physics? Probing Across Pretraining Paradigms](https://mlsanity.com/paper/2606.09646/) (2026)
- [The Escalator Problem: Identifying Implicit Motion Blindness in AI for Accessibility](https://mlsanity.com/paper/2508.07989/) (2025)
- [NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics](https://mlsanity.com/paper/2603.18761/) (2026)
- [Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs](https://mlsanity.com/paper/2608.07014/) (2026)
- [Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models](https://mlsanity.com/paper/2512.22249/) (2025)
- [RefereeBench: Are Video MLLMs Ready to be Multi-Sport Referees](https://mlsanity.com/paper/2604.15736/) (2026)
- [Incentivizing Temporal-Awareness in Egocentric Video Understanding Models](https://mlsanity.com/paper/2603.27184/) (2026)
- [BRITE: A Benchmark for Reliable and Interpretable T2V Evaluation on Implausible Scenarios](https://mlsanity.com/paper/2605.00873/) (2026)
- [VISTA: Video Interaction Spatio-Temporal Analysis Benchmark](https://mlsanity.com/paper/2605.01391/) (2026)
- [An Attribute-Based Measure of Video Complexity](https://mlsanity.com/paper/2606.00640/) (2026)
- [What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation](https://mlsanity.com/paper/2607.12304/) (2026)
- [ChronoVision: Temporal Reasoning via Latent State Reconstruction](https://mlsanity.com/paper/2608.05631/) (2026)
- [The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping](https://mlsanity.com/paper/2608.06361/) (2026)
- [Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study](https://mlsanity.com/paper/2608.17583/) (2026)
- [DementiaCare-Bench: A Modality-Validated Video Benchmark](https://mlsanity.com/paper/2609.12929/) (2026)
- [VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration](https://mlsanity.com/paper/2609.21521/) (2026)
- [Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models](https://mlsanity.com/paper/2609.28991/) (2026)
- [EventLens: Event-Structure Reinforcement Learning for Video Understanding](https://mlsanity.com/paper/2610.16172/) (2026)

## Abstract (arXiv)

Existing video understanding benchmarks often conflate knowledge-based and purely image-based questions, rather than clearly isolating a model's temporal reasoning ability, which is the key aspect that distinguishes video understanding from other modalities. We identify two major limitations that obscure whether higher scores truly indicate stronger understanding of the dynamic content in videos: (1) strong language priors, where models can answer questions without watching the video; and (2) shuffling invariance, where models maintain similar performance on certain questions even when video frames are temporally shuffled. To alleviate these issues, we propose VBenchComp, an automated pipeline that categorizes questions into different domains: LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions can be answered without viewing the video; Semantic questions remain answerable even when the video frames are shuffled; and Temporal questions require understanding the correct temporal order of frames. The rest of the questions are labeled as Others. This can enable fine-grained evaluation of different capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that are hidden by traditional overall scores, and we offer insights and recommendations for designing future benchmarks that more accurately assess video LLMs.