mlsanityData to 14 Oct 2026

A Comprehensive Study of Deep Video Action Recognition

arXiv 2012.06567 ↗First posted 11 Dec 2020cs.CV cs.MMcited by 88

Not on the map. Why?

Abstract (arXiv)

Video action recognition is one of the representative tasks for video understanding. Over the last decade, we have witnessed great advancements in video action recognition thanks to the emergence of deep learning. But we also encountered new challenges, including modeling long-range temporal information in videos, high computation costs, and incomparable results due to datasets and evaluation protocol variances. In this paper, we provide a comprehensive survey of over 200 existing papers on deep learning for video action recognition. We first introduce the 17 video action recognition datasets that influenced the design of models. Then we present video action recognition models in chronological order: starting with early attempts at adapting deep learning, then to the two-stream networks, followed by the adoption of 3D convolutional kernels, and finally to the recent compute-efficient models. In addition, we benchmark popular methods on several representative datasets and release code for reproducibility. In the end, we discuss open problems and shed light on opportunities for video action recognition to facilitate new research ideas.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 26; 2023: 23; 2024: 14; 2025: 10; 2026: 4.

88 papers.

Sort:

By area:

  1. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  2. 2023Video Understanding with Large Language Models: A Surveycited by 206
  3. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  4. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  5. 2022Physical Adversarial Attack meets Computer Vision: A Decade Surveycited by 67
  6. 2021Self-supervised Video Transformercited by 64
  7. 2021Video Contrastive Learning with Global Contextcited by 41
  8. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  9. 2021OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languagescited by 31
  10. 2022A Survey on Video Action Recognition in Sports: Datasets, Methods and Applicationscited by 28
  11. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  12. 2022A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Informationcited by 26
38 more
  1. 2023Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videoscited by 20
  2. 2024SignCLIP: Connecting Text and Sign Language by Contrastive Learningcited by 20
  3. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 20
  4. 2022Egocentric Video Task Translationcited by 19
  5. 2023AutoLabel: CLIP-based framework for Open-set Video Domain Adaptationcited by 18
  6. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  7. 2022Is Appearance Free Action Recognition Possible?cited by 16
  8. 2023The Unreasonable Effectiveness of Large Language-Vision Models for Source-free Video Domain Adaptationcited by 16
  9. 2025Impossible Videoscited by 16
  10. 2021Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reductioncited by 14
  11. 2024MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotationscited by 14
  12. 2021IntFormer: Predicting pedestrian intention with the aid of the Transformer architecturecited by 13
  13. 2022Sports Video Analysis on Large-Scale Datacited by 13
  14. 2023E2E-LOAD: End-to-End Long-form Online Action Detectioncited by 8
  15. 2024One missing piece in Vision and Language: A Survey on Comics Understandingcited by 8
  16. 2022Unified Recurrence Modeling for Video Action Anticipationcited by 7
  17. 2022End-to-end Transformer for Compressed Video Quality Enhancementcited by 7
  18. 2022Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networkscited by 7
  19. 2025Can masking background and object reduce static bias for zero-shot action recognition?cited by 6
  20. 2021TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decodingcited by 5
  21. 2022Pedestrian Stop and Go Forecasting with Hybrid Feature Fusioncited by 5
  22. 2022P2ANet: A Dataset and Benchmark for Dense Action Detection from Table Tennis Match Broadcasting Videoscited by 5
  23. 2023Video-based Contrastive Learning on Decision Trees: from Action Recognition to Autism Diagnosiscited by 5
  24. 2023Student Classroom Behavior Detection based on YOLOv7-BRA and Multi-Model Fusioncited by 5
  25. 2023SCB-Dataset3: A Benchmark for Detecting Student Classroom Behaviorcited by 5
  26. 2024BehAVE: Behaviour Alignment of Video Game Encodingscited by 5
  27. 2025The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognitioncited by 5
  28. 2021VidHarm: A Clip Based Dataset for Harmful Content Detectioncited by 4
  29. 2021Evaluating Transformers for Lightweight Action Recognitioncited by 4
  30. 2022ViGAT: Bottom-up event recognition and explanation in video using factorized graph attention networkcited by 4
  31. 2023CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Surveycited by 4
  32. 2023Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videoscited by 4
  33. 2024Attention Prompt Tuning: Parameter-efficient Adaptation of Pre-trained Models for Spatiotemporal Modelingcited by 4
  34. 2022Performance Evaluation of Action Recognition Models on Low Quality Videoscited by 3
  35. 2022Improving Transfer Learning with a Dual Image and Video Transformer for Multi-label Movie Trailer Genre Classificationcited by 3
  36. 2022Collaborative Reasoning on Multi-Modal Semantic Graphs for Video-Grounded Dialogue Generationcited by 3
  37. 2022Transformers in Action Recognition: A Review on Temporal Modelingcited by 3
  38. 2023Learning Human-Human Interactions in Images from Weak Textual Supervisioncited by 3

50 most cited of 88 All 88 (JSON)