# Real-Time End-to-End Action Detection with Two-Stream Networks

arXiv: https://arxiv.org/abs/1802.08362 · HTML: https://mlsanity.com/paper/1802.08362/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Feb 2018.

## Built on by (most cited first, 4 of 4)

- [Video Action Detection: Analysing Limitations and Challenges](https://mlsanity.com/paper/2204.07892/) (2022)
- [ACDnet: An action detection network for real-time edge computing based on flow-guided feature approximation and memory aggregation](https://mlsanity.com/paper/2102.13493/) (2021)
- [Exploiting Instance-based Mixed Sampling via Auxiliary Source Domain Supervision for Domain-adaptive Action Detection](https://mlsanity.com/paper/2209.15439/) (2022)
- [Conceptualizing Multi-scale Wavelet Attention and Ray-based Encoding for Human-Object Interaction Detection](https://mlsanity.com/paper/2507.10977/) (2025)

## Abstract (arXiv)

Two-stream networks have been very successful for solving the problem of action detection. However, prior work using two-stream networks train both streams separately, which prevents the network from exploiting regularities between the two streams. Moreover, unlike the visual stream, the dominant forms of optical flow computation typically do not maximally exploit GPU parallelism. We present a real-time end-to-end trainable two-stream network for action detection. First, we integrate the optical flow computation in our framework by using Flownet2. Second, we apply early fusion for the two streams and train the whole pipeline jointly end-to-end. Finally, for better network initialization, we transfer from the task of action recognition to action detection by pre-training our framework using the recently released large-scale Kinetics dataset. Our experimental results show that training the pipeline jointly end-to-end with fine-tuning the optical flow for the objective of action detection improves detection performance significantly. Additionally, we observe an improvement when initializing with parameters pre-trained using Kinetics. Last, we show that by integrating the optical flow computation, our framework is more efficient, running at real-time speeds (up to 31 fps).