mlsanityData to 7 Oct 2026

A Survey on Deep Learning-based Spatio-temporal Action Detection

arXiv 2308.01618 ↗First posted 3 Aug 2023cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Spatio-temporal action detection (STAD) aims to classify the actions present in a video and localize them in space and time. It has become a particularly active area of research in computer vision because of its explosively emerging real-world applications, such as autonomous driving, visual surveillance, entertainment, etc. Many efforts have been devoted in recent years to building a robust and effective framework for STAD. This paper provides a comprehensive review of the state-of-the-art deep learning-based methods for STAD. Firstly, a taxonomy is developed to organize these methods. Next, the linking algorithms, which aim to associate the frame- or clip-level detection results together to form action tubes, are reviewed. Then, the commonly used benchmark datasets and evaluation metrics are introduced, and the performance of state-of-the-art models is compared. At last, this paper is concluded, and a set of potential research directions of STAD are discussed.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 72 of its 103 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

72 of 103 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
38 more
  1. 2017Non-local Neural Networkscited by 2,290
  2. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2018SlowFast Networks for Video Recognitioncited by 1,862
  5. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  7. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  8. 2020Sparse R-CNN: End-to-End Object Detection with Learnable Proposalscited by 494
  9. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  10. 2017Deep Layer Aggregationcited by 436
  11. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  12. 2018Video Action Transformer Networkcited by 316
  13. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  14. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  15. 2015Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videoscited by 93
  16. 2020Actor-Context-Actor Relation Network for Spatio-Temporal Action Localizationcited by 91
  17. 2018Actor-Centric Relation Networkcited by 87
  18. 2021MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actionscited by 82
  19. 2017Action Tubelet Detector for Spatio-Temporal Action Localizationcited by 77
  20. 2023Diffusion Action Segmentationcited by 74
  21. 2020Asynchronous Interaction Aggregation for Action Detectioncited by 65
  22. 2019You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localizationcited by 58
  23. 2016Online Real-time Multiple Spatiotemporal Action Localisation and Predictioncited by 57
  24. 2020Actions as Moving Pointscited by 55
  25. 2021TubeR: Tubelet Transformer for Video Action Detectioncited by 55
  26. 2019Towards Segmenting Anything That Movescited by 51
  27. 2019STEP: Spatio-Temporal Progressive Learning for Video Action Detectioncited by 51
  28. 2014Finding Action Tubescited by 50
  29. 2017Tube Convolutional Neural Network (T-CNN) for Action Detection in Videoscited by 50
  30. 2015Learning to track for spatio-temporal action localizationcited by 44
  31. 2018A Structured Model For Action Detectioncited by 43
  32. 2019TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detectioncited by 36
  33. 2016Deep Learning for Detecting Multiple Space-Time Action Tubes in Videoscited by 35
  34. 2017Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detectioncited by 34
  35. 2020Context-Aware RCNN: A Baseline for Action Detection in Videoscited by 33
  36. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  37. 2018VideoCapsuleNet: A Simplified Network for Action Detectioncited by 30
  38. 2019Dance with Flow: Two-in-One Stream Action Detectioncited by 30

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Beyond Pixels: Leveraging the Language of Soccer to Improve Spatio-Temporal Action Detection in Broadcast Videoscited by 5
  2. 2025FOOTPASS: A Multi-Modal Multi-Agent Tactical Context Dataset for Play-by-Play Action Spotting in Soccer Broadcast Videoscited by 5
  3. 2024Human Stone Toolmaking Action Grammar (HSTAG): A Challenging Benchmark for Fine-grained Motor Behavior Recognitioncited by 0
  4. 2025Scaling Open-Vocabulary Action Detectioncited by 0
  5. 2026All Eyes on the Workflow: Automated and Efficient Event Discovery from Video Streamscited by 0