# You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization

arXiv: https://arxiv.org/abs/1911.06644 · HTML: https://mlsanity.com/paper/1911.06644/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2019.

## Built on by (most cited first, 30 of 59)

- [MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions](https://mlsanity.com/paper/2105.07404/) (2021)
- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2303.15879/) (2023)
- [Efficient Video Action Detection with Token Dropout and Context Refinement](https://mlsanity.com/paper/2304.08451/) (2023)
- [End-to-End Spatio-Temporal Action Localisation with Video Transformers](https://mlsanity.com/paper/2304.12160/) (2023)
- [Real-Time Idling Vehicles Detection using Combined Audio-Visual Deep Learning](https://mlsanity.com/paper/2305.14579/) (2023)
- [VA-RED2: Video Adaptive Redundancy Reduction](https://mlsanity.com/paper/2102.07887/) (2021)
- [Spatio-Temporal Action Detection Under Large Motion](https://mlsanity.com/paper/2209.02250/) (2022)
- [Joint Audio-Visual Idling Vehicle Detection with Streamlined Input Dependencies](https://mlsanity.com/paper/2410.21170/) (2024)
- [Continual 3D Convolutional Neural Networks for Real-time Processing of Videos](https://mlsanity.com/paper/2106.00050/) (2021)
- [SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning](https://mlsanity.com/paper/2504.00527/) (2025)
- [Beyond Transfer Learning: Co-finetuning for Action Localisation](https://mlsanity.com/paper/2207.03807/) (2022)
- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [Survey of Action Recognition, Spotting and Spatio-Temporal Localization in Soccer -- Current Trends and Research Perspectives](https://mlsanity.com/paper/2309.12067/) (2023)
- [ACDnet: An action detection network for real-time edge computing based on flow-guided feature approximation and memory aggregation](https://mlsanity.com/paper/2102.13493/) (2021)
- [Multi-scale Context-aware Network with Transformer for Gait Recognition](https://mlsanity.com/paper/2204.03270/) (2022)
- [A Simple and Efficient Pipeline to Build an End-to-End Spatial-Temporal Action Detector](https://mlsanity.com/paper/2206.03064/) (2022)
- [YOWOv2: A Stronger yet Efficient Multi-level Detection Framework for Real-time Spatio-temporal Action Detection](https://mlsanity.com/paper/2302.06848/) (2023)
- [CathAction: A Benchmark for Endovascular Intervention Understanding](https://mlsanity.com/paper/2408.13126/) (2024)
- [On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes](https://mlsanity.com/paper/2410.19553/) (2024)
- [Stable Mean Teacher for Semi-supervised Video Action Detection](https://mlsanity.com/paper/2412.07072/) (2024)
- [Game State and Spatio-temporal Action Detection in Soccer using Graph Neural Networks and 3D Convolutional Networks](https://mlsanity.com/paper/2502.15462/) (2025)
- [MOVE: Motion-Guided Few-Shot Video Object Segmentation](https://mlsanity.com/paper/2507.22061/) (2025)
- [Interaction-Aware Prompting for Zero-Shot Spatio-Temporal Action Detection](https://mlsanity.com/paper/2304.04688/) (2023)
- [MRSN: Multi-Relation Support Network for Video Action Detection](https://mlsanity.com/paper/2304.11975/) (2023)
- [A Survey on Deep Learning-based Spatio-temporal Action Detection](https://mlsanity.com/paper/2308.01618/) (2023)
- [Open-Vocabulary Spatio-Temporal Action Detection](https://mlsanity.com/paper/2405.10832/) (2024)
- [Multiscale Vision Transformers meet Bipartite Matching for efficient single-stage Action Localization](https://mlsanity.com/paper/2312.17686/) (2023)
- [Temp-Frustum Net: 3D Object Detection with Temporal Fusion](https://mlsanity.com/paper/2104.12106/) (2021)

## Abstract (arXiv)

Spatiotemporal action localization requires the incorporation of two sources of information into the designed architecture: (1) temporal information from the previous frames and (2) spatial information from the key frame. Current state-of-the-art approaches usually extract these information with separate networks and use an extra mechanism for fusion to get detections. In this work, we present YOWO, a unified CNN architecture for real-time spatiotemporal action localization in video streams. YOWO is a single-stage architecture with two branches to extract temporal and spatial information concurrently and predict bounding boxes and action probabilities directly from video clips in one evaluation. Since the whole architecture is unified, it can be optimized end-to-end. The YOWO architecture is fast providing 34 frames-per-second on 16-frames input clips and 62 frames-per-second on 8-frames input clips, which is currently the fastest state-of-the-art architecture on spatiotemporal action localization task. Remarkably, YOWO outperforms the previous state-of-the art results on J-HMDB-21 and UCF101-24 with an impressive improvement of ~3% and ~12%, respectively. Moreover, YOWO is the first and only single-stage architecture that provides competitive results on AVA dataset. We make our code and pretrained models publicly available.