mlsanityData to 7 Oct 2026

Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives

arXiv 2406.05615 ↗First posted 9 Jun 2024cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Humans use multiple senses to comprehend the environment. Vision and language are two of the most vital senses since they allow us to easily communicate our thoughts and perceive the world around us. There has been a lot of interest in creating video-language understanding systems with human-like senses since a video-language pair can mimic both our linguistic medium and visual environment with temporal dynamics. In this survey, we review the key tasks of these systems and highlight the associated challenges. Based on the challenges, we summarize their methods from model architecture, model training, and data perspectives. We also conduct performance comparison among the methods, and discuss promising directions for future research.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 128 of its 180 references that are in the map. First posted on arXiv 9 Jun 2024.

Builds on

128 of 180 references.

References · 128

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
38 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  8. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  9. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  10. 2020Rescaling Egocentric Visioncited by 529
  11. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  13. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  15. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  16. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  17. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  18. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  19. 2018A Short Note about Kinetics-600cited by 331
  20. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  21. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  22. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  23. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  24. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  25. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  26. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  27. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  28. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  29. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  30. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  31. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  32. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  33. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  34. 2023Video Understanding with Large Language Models: A Surveycited by 203
  35. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  36. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  37. 2015A Dataset for Movie Descriptioncited by 195
  38. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 16.

37 papers.

Sort:

By area:

  1. 2024GalleryGPT: Analyzing Paintings with Large Multimodal Modelscited by 28
  2. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  3. 2024Do Language Models Understand Time?cited by 14
  4. 2025MVP: Winning Solution to SMP Challenge 2025 Video Trackcited by 14
  5. 2025Agentic Keyframe Search for Video Question Answeringcited by 13
  6. 2024Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generationcited by 9
  7. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  8. 2024MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learningcited by 8
  9. 2025TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systemscited by 8
  10. 2026ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understandingcited by 7
  11. 2026CurEvo: Curriculum-Guided Self-Evolution for Video Understandingcited by 3
  12. 2025MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Modelscited by 2
25 more
  1. 2025Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Groundingcited by 2
  2. 2025What Happens When: Learning Temporal Orders of Events in Videoscited by 2
  3. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  4. 2026VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inferencecited by 2
  5. 2024Multimodal Fusion and Coherence Modeling for Video Topic Segmentationcited by 1
  6. 2024Topic-aware Causal Intervention for Counterfactual Detectioncited by 1
  7. 2025Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understandingcited by 1
  8. 2025Leveraging Auxiliary Information in Text-to-Video Retrieval: A Reviewcited by 1
  9. 2025Do We Need Large VLMs for Spotting Soccer Actions?cited by 1
  10. 2025Iterative Zoom-In: Temporal Interval Exploration for Long Video Understandingcited by 1
  11. 2025Vote-in-Context: Turning VLMs into Zero-Shot Rank Fuserscited by 1
  12. 2025More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answeringcited by 1
  13. 2026Process-of-Thought Reasoning for Videoscited by 1
  14. 2025Video Understanding by Design: How Datasets Shape Video Modelscited by 0
  15. 2026PREGEN: Uncovering Latent Thoughts in Composed Video Retrievalcited by 0
  16. 2026StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Datasetcited by 0
  17. 2026Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extractioncited by 0
  18. 2026Empowering Long-form Omni-modal Understanding with Robust Audio Perceptioncited by 0
  19. 2026Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videoscited by 0
  20. 2026EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibrationcited by 0
  21. 2026GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Modelscited by 0
  22. 2026LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agentscited by 0
  23. 2026RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understandingcited by 0
  24. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0
  25. 2026Agentic Video Understanding: A Surveycited by 0