mlsanityData to 14 Oct 2026

TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

arXiv 2410.10818 ↗First posted 14 Oct 2024cs.CV cs.AI cs.CLcited by 97

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding fine-grained temporal dynamics is crucial for multimodal video comprehension and generation. Due to the lack of fine-grained temporal annotations, existing video benchmarks mostly resemble static image benchmarks and are incompetent at evaluating models for temporal understanding. In this paper, we introduce TemporalBench, a new benchmark dedicated to evaluating fine-grained temporal understanding in videos. TemporalBench consists of ~10K video question-answer pairs, derived from ~2K high-quality human annotations detailing the temporal dynamics in video clips. As a result, our benchmark provides a unique testbed for evaluating various temporal understanding and reasoning abilities such as action frequency, motion magnitude, event order, etc. Moreover, it enables evaluations on various tasks like both video question answering and captioning, both short and long video understanding, as well as different models such as multimodal video embedding models and text generation models. Results show that state-of-the-art models like GPT-4o achieve only 38.5% question answering accuracy on TemporalBench, demonstrating a significant gap (~30%) between humans and AI in temporal understanding. Furthermore, we notice a critical pitfall for multi-choice QA where LLMs can detect the subtle changes in negative captions and find a centralized description as a cue for its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such bias. We hope that TemporalBench can foster research on improving models' temporal reasoning capabilities. Both dataset and evaluation code will be made available.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Builds on 64 of its 83 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

64 of 83 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  14. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  15. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  17. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  18. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  21. 2017Dense-Captioning Events in Videoscited by 793
  22. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  23. 2024Yi: Open Foundation Models by 01.AIcited by 719
  24. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  25. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  26. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  27. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  28. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  29. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  30. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  31. 2021InfographicVQAcited by 437
  32. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  33. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  34. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  35. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  36. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  37. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  38. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 46; 2026: 44.

97 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  2. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  3. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  4. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  5. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 114
  6. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  7. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  8. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 61
  9. 2026MiniMax Sparse Attentioncited by 44
  10. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  11. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 36
  12. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
38 more
  1. 2025MINERVA: Evaluating Complex Video Reasoningcited by 32
  2. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 30
  3. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 30
  4. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  5. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
  6. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 27
  7. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  8. 2025Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understandingcited by 24
  9. 2025VideoGen-Eval: Agent-based System for Video Generation Evaluationcited by 23
  10. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 21
  11. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 20
  12. 2025Fostering Video Reasoning via Next-Event Predictioncited by 18
  13. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 18
  14. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!cited by 15
  15. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  16. 2025AdsQA: Towards Advertisement Video Understandingcited by 13
  17. 2025MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Groundingcited by 12
  18. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 12
  19. 2024Progress-Aware Video Frame Captioningcited by 11
  20. 2025Video SimpleQA: Towards Factuality Evaluation in Large Video Language Modelscited by 11
  21. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  22. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  23. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  24. 2025Causality Matters: How Temporal Information Emerges in Video Language Modelscited by 10
  25. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  26. 2025CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videoscited by 9
  27. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  28. 2025GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?cited by 8
  29. 2025PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assemblycited by 7
  30. 2026IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streamscited by 7
  31. 2025VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Modelscited by 5
  32. 2025From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervisioncited by 4
  33. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
  34. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 4
  35. 2026FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understandingcited by 4
  36. 2025LLaVAction: evaluating and training multi-modal large language models for action understandingcited by 3
  37. 2025TempCore: Are Video QA Benchmarks Temporally Grounded?cited by 3
  38. 2025NeMo: Needle in a Montage for Video-Language Understandingcited by 3

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  2. 2025TextVidBench: A Benchmark for Long Video Scene Text Understanding
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  4. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  5. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
  6. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding
  7. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding
  8. 2024HourVideo: 1-Hour Video-Language Understanding