mlsanityData to 7 Oct 2026

Towards Long-Form Video Understanding

arXiv 2106.11310 ↗First posted 21 Jun 2021cs.CVcited by 146

Not on the map. Why?

Abstract (arXiv)

Our world offers a never-ending stream of visual stimuli, yet today's vision systems only accurately recognize patterns within a few seconds. These systems understand the present, but fail to contextualize it in past or future events. In this paper, we study long-form video understanding. We introduce a framework for modeling long-form videos and develop evaluation protocols on large-scale datasets. We show that existing state-of-the-art short-term models are limited for long-form tasks. A novel object-centric transformer-based video recognition architecture performs significantly better on 7 diverse tasks. It also outperforms comparable state-of-the-art on the AVA dataset.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 73 of its 93 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

73 of 93 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2017Non-local Neural Networkscited by 2,290
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  13. 2019Contrastive Multiview Codingcited by 1,297
  14. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  17. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  18. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  19. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  20. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  21. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  22. 2017Look, Listen and Learncited by 452
  23. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  24. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  25. 2018A Short Note about Kinetics-600cited by 331
  26. 2017Temporal Relational Reasoning in Videoscited by 324
  27. 2018Video Action Transformer Networkcited by 316
  28. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  29. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  30. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  31. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  32. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  33. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  34. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  35. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  36. 2018Videos as Space-Time Region Graphscited by 221
  37. 2015Learning Multi-Domain Convolutional Neural Networks for Visual Trackingcited by 209
  38. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 30; 2023: 34; 2024: 40; 2025: 25; 2026: 11.

146 papers.

Sort:

By area:

  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  3. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  4. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  5. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  6. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  7. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  8. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  9. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  10. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  11. 2022Vision-Centric BEV Perception: A Surveycited by 126
  12. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
38 more
  1. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  2. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  3. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  4. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  5. 2021Long Short-Term Transformer for Online Action Detectioncited by 87
  6. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  7. 2024Streaming Dense Video Captioningcited by 82
  8. 2021Object-Region Video Transformerscited by 73
  9. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  10. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  11. 2022Video Transformers: A Surveycited by 63
  12. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  13. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  14. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  15. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  16. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  17. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  18. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  19. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  20. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  21. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  22. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  23. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  24. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  25. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  26. 2023On the Benefits of 3D Pose and Tracking for Human Action Recognitioncited by 32
  27. 2022Holistic Interaction Transformer Network for Action Detectioncited by 28
  28. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  29. 2022CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Groundingcited by 27
  30. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25
  31. 2024VideoMamba: Spatio-Temporal Selective State Space Modelcited by 25
  32. 2022The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editingcited by 24
  33. 2023Object-centric Video Representation for Long-term Action Anticipationcited by 24
  34. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 24
  35. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  36. 2021Exploring Segment-level Semantics for Online Phase Recognition from Surgical Videoscited by 23
  37. 2022EgoEnv: Human-centric environment representations from egocentric videocited by 22
  38. 2024Video Token Merging for Long-form Video Understandingcited by 22

50 most cited of 146 All 146 (JSON)