# Temporally smooth online action detection using cycle-consistent future anticipation

arXiv: https://arxiv.org/abs/2104.08030 · HTML: https://mlsanity.com/paper/2104.08030/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos](https://mlsanity.com/paper/1507.05738/) (2015)
- [W-TALC: Weakly-supervised Temporal Activity Localization and Classification](https://mlsanity.com/paper/1807.10418/) (2018)
- [Temporal Recurrent Networks for Online Action Detection](https://mlsanity.com/paper/1811.07391/) (2018)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [AutoLoc: Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/1807.08333/) (2018)
- [Online Action Detection](https://mlsanity.com/paper/1604.06506/) (2016)
- [CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016](https://mlsanity.com/paper/1608.00797/) (2016)
- [Predicting Deeper into the Future of Semantic Segmentation](https://mlsanity.com/paper/1703.07684/) (2017)
- [Predicting the Future: A Jointly Learnt Model for Action Anticipation](https://mlsanity.com/paper/1912.07148/) (2019)
- [Looking Ahead: Anticipating Pedestrians Crossing with Future Frames Prediction](https://mlsanity.com/paper/1910.09077/) (2019)
- [Im2Flow: Motion Hallucination from Static Images for Action Recognition](https://mlsanity.com/paper/1712.04109/) (2017)
- [StartNet: Online Detection of Action Start in Untrimmed Videos](https://mlsanity.com/paper/1903.09868/) (2019)
- [Action Anticipation By Predicting Future Dynamic Images](https://mlsanity.com/paper/1808.00141/) (2018)
- [Online Detection of Action Start in Untrimmed, Streaming Videos](https://mlsanity.com/paper/1802.06822/) (2018)
- [Early Action Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1904.13085/) (2019)

## Built on by (most cited first, 13 of 13)

- [Long Short-Term Transformer for Online Action Detection](https://mlsanity.com/paper/2107.03377/) (2021)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [GateHUB: Gated History Unit with Background Suppression for Online Action Detection](https://mlsanity.com/paper/2206.04668/) (2022)
- [Context-Enhanced Memory-Refined Transformer for Online Action Detection](https://mlsanity.com/paper/2503.18359/) (2025)
- [Continual Transformers: Redundancy-Free Attention for Online Inference](https://mlsanity.com/paper/2201.06268/) (2022)
- [HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization](https://mlsanity.com/paper/2408.06437/) (2024)
- [E2E-LOAD: End-to-End Long-form Online Action Detection](https://mlsanity.com/paper/2306.07703/) (2023)
- [JOADAA: joint online action detection and action anticipation](https://mlsanity.com/paper/2309.06130/) (2023)
- [A Circular Window-based Cascade Transformer for Online Action Detection](https://mlsanity.com/paper/2208.14209/) (2022)
- [ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos](https://mlsanity.com/paper/2407.12987/) (2024)
- [Efficient Online Processing with Deep Neural Networks](https://mlsanity.com/paper/2306.13474/) (2023)
- [Probabilistic Temporal Masked Attention for Cross-view Online Action Detection](https://mlsanity.com/paper/2508.17025/) (2025)
- [CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning](https://mlsanity.com/paper/2603.23988/) (2026)

## Abstract (arXiv)

Many video understanding tasks work in the offline setting by assuming that the input video is given from the start to the end. However, many real-world problems require the online setting, making a decision immediately using only the current and the past frames of videos such as in autonomous driving and surveillance systems. In this paper, we present a novel solution for online action detection by using a simple yet effective RNN-based networks called the Future Anticipation and Temporally Smoothing network (FATSnet). The proposed network consists of a module for anticipating the future that can be trained in an unsupervised manner with the cycle-consistency loss, and another component for aggregating the past and the future for temporally smooth frame-by-frame predictions. We also propose a solution to relieve the performance loss when running RNN-based models on very long sequences. Evaluations on TVSeries, THUMOS14, and BBDB show that our method achieve the state-of-the-art performances compared to the previous works on online action detection.