# StartNet: Online Detection of Action Start in Untrimmed Videos

arXiv: https://arxiv.org/abs/1903.09868 · HTML: https://mlsanity.com/paper/1903.09868/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2019.

## Built on by (most cited first, 21 of 21)

- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [OadTR: Online Action Detection with Transformers](https://mlsanity.com/paper/2106.11149/) (2021)
- [Real-time Online Video Detection with Temporal Smoothing Transformers](https://mlsanity.com/paper/2209.09236/) (2022)
- [Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding](https://mlsanity.com/paper/2504.13915/) (2025)
- [Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild](https://mlsanity.com/paper/2103.10391/) (2021)
- [Coupling Intent and Action for Pedestrian Crossing Behavior Prediction](https://mlsanity.com/paper/2105.04133/) (2021)
- [Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models](https://mlsanity.com/paper/2203.16755/) (2022)
- [Temporally smooth online action detection using cycle-consistent future anticipation](https://mlsanity.com/paper/2104.08030/) (2021)
- [Context-Enhanced Memory-Refined Transformer for Online Action Detection](https://mlsanity.com/paper/2503.18359/) (2025)
- [Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos](https://mlsanity.com/paper/2203.13309/) (2022)
- [ENIGMA-51: Towards a Fine-Grained Understanding of Human-Object Interactions in Industrial Scenarios](https://mlsanity.com/paper/2309.14809/) (2023)
- [HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization](https://mlsanity.com/paper/2408.06437/) (2024)
- [E2E-LOAD: End-to-End Long-form Online Action Detection](https://mlsanity.com/paper/2306.07703/) (2023)
- [Online Temporal Action Localization with Memory-Augmented Transformer](https://mlsanity.com/paper/2408.02957/) (2024)
- [Untrimmed Action Anticipation](https://mlsanity.com/paper/2202.04132/) (2022)
- [SimOn: A Simple Framework for Online Temporal Action Localization](https://mlsanity.com/paper/2211.04905/) (2022)
- [JOADAA: joint online action detection and action anticipation](https://mlsanity.com/paper/2309.06130/) (2023)
- [ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos](https://mlsanity.com/paper/2407.12987/) (2024)
- [Streaming Detection of Queried Event Start](https://mlsanity.com/paper/2412.03567/) (2024)
- [Mamba-OTR: a Mamba-based Solution for Online Take and Release Detection from Untrimmed Egocentric Video](https://mlsanity.com/paper/2507.16342/) (2025)
- [Online Misinformation Detection in Live Streaming Videos](https://mlsanity.com/paper/2503.12627/) (2025)

## Abstract (arXiv)

We propose StartNet to address Online Detection of Action Start (ODAS) where action starts and their associated categories are detected in untrimmed, streaming videos. Previous methods aim to localize action starts by learning feature representations that can directly separate the start point from its preceding background. It is challenging due to the subtle appearance difference near the action starts and the lack of training data. Instead, StartNet decomposes ODAS into two stages: action classification (using ClsNet) and start point localization (using LocNet). ClsNet focuses on per-frame labeling and predicts action score distributions online. Based on the predicted action scores of the past and current frames, LocNet conducts class-agnostic start detection by optimizing long-term localization rewards using policy gradient methods. The proposed framework is validated on two large-scale datasets, THUMOS'14 and ActivityNet. The experimental results show that StartNet significantly outperforms the state-of-the-art by 15%-30% p-mAP under the offset tolerance of 1-10 seconds on THUMOS'14, and achieves comparable performance on ActivityNet with 10 times smaller time offset.