mlsanityData to 7 Oct 2026

VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models

arXiv 2311.17404 ↗First posted 29 Nov 2023cs.CV cs.AI cs.CLcited by 55

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to perceive how objects change over time is a crucial ingredient in human intelligence. However, current benchmarks cannot faithfully reflect the temporal understanding abilities of video-language models (VidLMs) due to the existence of static visual shortcuts. To remedy this issue, we present VITATECS, a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding. Specifically, we first introduce a fine-grained taxonomy of temporal concepts in natural language in order to diagnose the capability of VidLMs to comprehend different temporal aspects. Furthermore, to disentangle the correlation between static and temporal information, we generate counterfactual video descriptions that differ from the original one only in the specified temporal aspect. We employ a semi-automatic data collection framework using large language models and human-in-the-loop annotation to obtain high-quality counterfactual descriptions efficiently. Evaluation of representative video-language understanding models confirms their deficiency in temporal understanding, revealing the need for greater emphasis on the temporal elements in video-language research.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 78 of its 106 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

78 of 106 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  13. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  14. 2021Video Swin Transformercited by 1,090
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  16. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  17. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  18. 2017Dense-Captioning Events in Videoscited by 789
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  20. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  21. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  22. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  23. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  24. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  25. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  26. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  27. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  28. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  29. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  30. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  31. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  32. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  33. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 232
  34. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  35. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  36. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  37. 2015A Dataset for Movie Descriptioncited by 195
  38. 2021Generating Datasets with Pretrained Language Modelscited by 192

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 27; 2026: 10.

55 papers.

Sort:

By area:

  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  3. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  4. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  5. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  6. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  7. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  8. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  9. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  10. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  11. 2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
  12. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 37
38 more
  1. 2024Temporal Reasoning Transfer from Text to Videocited by 36
  2. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  3. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  4. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  5. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  7. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 24
  8. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  9. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  10. 2024On the Consistency of Video Large Language Models in Temporal Comprehensioncited by 18
  11. 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?cited by 18
  12. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  13. 2025Fostering Video Reasoning via Next-Event Predictioncited by 15
  14. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  15. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  16. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  17. 2024VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailmentcited by 10
  18. 2024Progress-Aware Video Frame Captioningcited by 10
  19. 2025TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Modelscited by 9
  20. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  21. 2025Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluationcited by 6
  22. 2025Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoderscited by 6
  23. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligencecited by 4
  24. 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectoriescited by 4
  25. 2025Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoningcited by 4
  26. 2024MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialoguescited by 3
  27. 2025TEMPLE: Incentivizing Temporal Understanding of Video Large Language Models via Progressive Pre-SFT Alignmentcited by 3
  28. 2025JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Roboticscited by 3
  29. 2026YoCausal: How Far is Video Generation from World Model? A Causality Perspectivecited by 3
  30. 2025H2VU-Benchmark: A Comprehensive Benchmark for Hierarchical Holistic Video Understandingcited by 2
  31. 2025What Happens When: Learning Temporal Orders of Events in Videoscited by 2
  32. 2026Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMscited by 2
  33. 2025RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Modelscited by 1
  34. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  35. 2026MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoningcited by 1
  36. 2026Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMscited by 1
  37. 2026AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Modelscited by 1
  38. 2025Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMscited by 0

50 most cited of 55 All 55 (JSON)