mlsanityData to 14 Oct 2026

Revisiting the "Video" in Video-Language Understanding

arXiv 2206.01720 ↗First posted 3 Jun 2022cs.CV cs.AI cs.CLcited by 169

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

What makes a video task uniquely suited for videos, beyond what can be understood from a single image? Building on recent progress in self-supervised image-language models, we revisit this question in the context of video and language tasks. We propose the atemporal probe (ATP), a new model for video-language analysis which provides a stronger bound on the baseline accuracy of multimodal models constrained by image-level understanding. By applying this model to standard discriminative video and language tasks, such as video question answering and text-to-video retrieval, we characterize the limitations and potential of current video-language benchmarks. We find that understanding of event temporality is often not necessary to achieve strong or state-of-the-art performance, even compared with recent large-scale video-language models and in contexts intended to benchmark deeper video-level understanding. We also demonstrate how ATP can improve both video-language dataset and model design. We describe a technique for leveraging ATP to better disentangle dataset subsets with a higher concentration of temporally challenging data, improving benchmarking efficacy for causal and temporal understanding. Further, we show that effectively integrating ATP into full video-level temporal models can improve efficiency and state-of-the-art accuracy.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

52 of 64 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2018SlowFast Networks for Video Recognitioncited by 1,867
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  6. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  8. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  9. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  10. 2017Dense-Captioning Events in Videoscited by 793
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  13. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  14. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  15. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  16. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  17. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  18. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  19. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  20. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  21. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  22. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  23. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  24. 2017Temporal Relational Reasoning in Videoscited by 324
  25. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  26. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  27. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  28. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  29. 2015A Dataset for Movie Descriptioncited by 195
  30. 2020Support-set bottlenecks for video-text representation learningcited by 170
  31. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  32. 2018AdaFrame: Adaptive Frame Selection for Fast Video Recognitioncited by 134
  33. 2015End-to-end Learning of Action Detection from Frame Glimpses in Videoscited by 101
  34. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  35. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 92
  36. 2018Localizing Moments in Video with Temporal Languagecited by 83
  37. 2020What is More Likely to Happen Next? Video-and-Language Future Event Predictioncited by 62
  38. 2021No frame left behind: Full Video Action Recognitioncited by 24

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 15; 2023: 47; 2024: 47; 2025: 37; 2026: 23.

169 papers.

Sort:

By area:

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  3. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  5. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  6. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  7. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  8. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  9. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  10. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  11. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  12. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
38 more
  1. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  2. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  3. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  4. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  5. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  6. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  7. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  8. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  9. 2022Video Graph Transformer for Video Question Answeringcited by 63
  10. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  11. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  12. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  13. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  14. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  15. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  16. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  17. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  18. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 46
  19. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  20. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  21. 2023Dual-path Adaptation from Image to Video Transformerscited by 41
  22. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  23. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answeringcited by 40
  24. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 38
  25. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  26. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  27. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  28. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 30
  29. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 30
  30. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  31. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  32. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 29
  33. 2023GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generationcited by 28
  34. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  35. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 25
  36. 2024LVD-2M: A Long-take Video Dataset with Temporally Dense Captionscited by 25
  37. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  38. 2022Language-free Training for Zero-shot Video Groundingcited by 22

50 most cited of 169 All 169 (JSON)