mlsanityData to 7 Oct 2026

TempCompass: Do Video LLMs Really Understand Videos?

arXiv 2403.00476 ↗First posted 1 Mar 2024cs.CVcited by 352

Not on the map. Why?

Cited by 163 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, there is a surge in interest surrounding video large language models (Video LLMs). However, existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs. On the one hand, most of them are unable to distinguish between different temporal aspects (e.g., speed, direction) and thus cannot reflect the nuanced performance on these specific aspects. On the other hand, they are limited in the diversity of task formats (e.g., only multi-choice QA), which hinders the understanding of how temporal perception performance may vary across different types of tasks. Motivated by these two problems, we propose the TempCompass benchmark, which introduces a diversity of temporal aspects and task formats. To collect high-quality test data, we devise two novel strategies: (1) In video collection, we construct conflicting videos that share the same static content but differ in a specific temporal aspect, which prevents Video LLMs from leveraging single-frame bias or language priors. (2) To collect the task instructions, we propose a paradigm where humans first annotate meta-information for a video and then an LLM generates the instruction. We also design an LLM-based approach to automatically and accurately evaluate the responses from Video LLMs. Based on TempCompass, we comprehensively evaluate 8 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs, and reveal the discerning fact that these models exhibit notably poor temporal perception ability. Our data will be available at https://github.com/llyx97/TempCompass.

Key facts. Cited by 352 papers in the map (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

27 of 38 references.

References · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
15 more
  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  2. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  4. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  5. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  6. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  8. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  9. 2022UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detectioncited by 148
  10. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  11. 2018Localizing Moments in Video with Temporal Languagecited by 83
  12. 2019Only Time Can Tell: Discovering Temporal Data for Temporal Modelingcited by 66
  13. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  14. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  15. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 60; 2025: 157; 2026: 135.

352 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  8. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  9. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  10. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  11. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  12. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
38 more
  1. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  2. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  3. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  4. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  5. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  6. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  8. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  9. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  10. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  11. 2025Ovis2.5 Technical Reportcited by 103
  12. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
  13. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  14. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  15. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
  16. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  17. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  18. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  19. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  20. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  21. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  22. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  23. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  24. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  25. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  26. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  27. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  28. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  29. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  30. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  31. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  32. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  33. 2025ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understandingcited by 50
  34. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  35. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  36. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  37. 2025Kwai Keye-VL Technical Reportcited by 47
  38. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47

50 most cited of 352 All 352 (JSON)