# Action Tubelet Detector for Spatio-Temporal Action Localization

arXiv: https://arxiv.org/abs/1705.01861 · HTML: https://mlsanity.com/paper/1705.01861/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2017.

## Built on by (most cited first, 30 of 77)

- [Relaxed Transformer Decoders for Direct Action Proposal Generation](https://mlsanity.com/paper/2102.01894/) (2021)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [TubeR: Tubelet Transformer for Video Action Detection](https://mlsanity.com/paper/2104.00969/) (2021)
- [ROAD: The ROad event Awareness Dataset for Autonomous Driving](https://mlsanity.com/paper/2102.11585/) (2021)
- [Activity Graph Transformer for Temporal Action Localization](https://mlsanity.com/paper/2101.08540/) (2021)
- [RGB-Event Fusion for Moving Object Detection in Autonomous Driving](https://mlsanity.com/paper/2209.08323/) (2022)
- [Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts](https://mlsanity.com/paper/2408.05905/) (2024)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods](https://mlsanity.com/paper/2104.03178/) (2021)
- [ROAD-R: The Autonomous Driving Dataset with Logical Requirements](https://mlsanity.com/paper/2210.01597/) (2022)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering](https://mlsanity.com/paper/2106.13432/) (2021)
- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [End-to-End Spatio-Temporal Action Localisation with Video Transformers](https://mlsanity.com/paper/2304.12160/) (2023)
- [Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos](https://mlsanity.com/paper/2110.10596/) (2021)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [SportsHHI: A Dataset for Human-Human Interaction Detection in Sports Videos](https://mlsanity.com/paper/2404.04565/) (2024)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data](https://mlsanity.com/paper/2212.04821/) (2022)
- [Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection](https://mlsanity.com/paper/2203.00307/) (2022)
- [Video Action Detection: Analysing Limitations and Challenges](https://mlsanity.com/paper/2204.07892/) (2022)
- [Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization](https://mlsanity.com/paper/2303.11003/) (2023)
- [Video Dialog as Conversation about Objects Living in Space-Time](https://mlsanity.com/paper/2207.03656/) (2022)
- [What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions](https://mlsanity.com/paper/2303.16990/) (2023)
- [Semi-supervised Active Learning for Video Action Detection](https://mlsanity.com/paper/2312.07169/) (2023)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [Continual 3D Convolutional Neural Networks for Real-time Processing of Videos](https://mlsanity.com/paper/2106.00050/) (2021)

## Abstract (arXiv)

Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level that are then linked or tracked across time. In this paper, we leverage the temporal continuity of videos instead of operating at the frame level. We propose the ACtion Tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs tubelets, i.e., sequences of bounding boxes with associated scores. The same way state-of-the-art object detectors rely on anchor boxes, our ACT-detector is based on anchor cuboids. We build upon the SSD framework. Convolutional features are extracted for each frame, while scores and regressions are based on the temporal stacking of these features, thus exploiting information from a sequence. Our experimental results show that leveraging sequences of frames significantly improves detection performance over using individual frames. The gain of our tubelet detector can be explained by both more accurate scores and more precise localization. Our ACT-detector outperforms the state-of-the-art methods for frame-mAP and video-mAP on the J-HMDB and UCF-101 datasets, in particular at high overlap thresholds.