# RGB Stream Is Enough for Temporal Action Detection

arXiv: https://arxiv.org/abs/2107.04362 · HTML: https://mlsanity.com/paper/2107.04362/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 61 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 30 of 48)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [G-TAD: Sub-Graph Localization for Temporal Action Detection](https://mlsanity.com/paper/1911.11462/) (2019)
- [Graph Convolutional Networks for Temporal Action Localization](https://mlsanity.com/paper/1909.03252/) (2019)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [R-C3D: Region Convolutional 3D Network for Temporal Activity Detection](https://mlsanity.com/paper/1703.07814/) (2017)
- [Learning Salient Boundary Feature for Anchor-free Temporal Action Localization](https://mlsanity.com/paper/2103.13137/) (2021)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)

## Built on by (most cited first, 16 of 16)

- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [TALLFormer: Temporal Action Localization with a Long-memory Transformer](https://mlsanity.com/paper/2204.01680/) (2022)
- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection](https://mlsanity.com/paper/2205.02717/) (2022)
- [Colar: Effective and Efficient Online Action Detection by Consulting Exemplars](https://mlsanity.com/paper/2203.01057/) (2022)
- [MMAD: Multi-label Micro-Action Detection in Videos](https://mlsanity.com/paper/2407.05311/) (2024)
- [Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization](https://mlsanity.com/paper/2211.14053/) (2022)
- [An Efficient Spatio-Temporal Pyramid Transformer for Action Detection](https://mlsanity.com/paper/2207.10448/) (2022)
- [Adapting Short-Term Transformers for Action Detection in Untrimmed Videos](https://mlsanity.com/paper/2312.01897/) (2023)
- [Stochastic Backpropagation: A Memory Efficient Strategy for Training Video Models](https://mlsanity.com/paper/2203.16755/) (2022)
- [Boundary-Denoising for Video Activity Localization](https://mlsanity.com/paper/2304.02934/) (2023)
- [Towards High-Quality Temporal Action Detection with Sparse Proposals](https://mlsanity.com/paper/2109.08847/) (2021)
- [Temporal Action Localization with Multi-temporal Scales](https://mlsanity.com/paper/2208.07493/) (2022)
- [Temporal2Seq: A Unified Framework for Temporal Video Understanding Tasks](https://mlsanity.com/paper/2409.18478/) (2024)
- [Facial Tic Detection in Untrimmed Videos of Tourette Syndrome Patients](https://mlsanity.com/paper/2211.03895/) (2022)

## Abstract (arXiv)

State-of-the-art temporal action detectors to date are based on two-stream input including RGB frames and optical flow. Although combining RGB frames and optical flow boosts performance significantly, optical flow is a hand-designed representation which not only requires heavy computation, but also makes it methodologically unsatisfactory that two-stream methods are often not learned end-to-end jointly with the flow. In this paper, we argue that optical flow is dispensable in high-accuracy temporal action detection and image level data augmentation (ILDA) is the key solution to avoid performance degradation when optical flow is removed. To evaluate the effectiveness of ILDA, we design a simple yet efficient one-stage temporal action detector based on single RGB stream named DaoTAD. Our results show that when trained with ILDA, DaoTAD has comparable accuracy with all existing state-of-the-art two-stream detectors while surpassing the inference speed of previous methods by a large margin and the inference speed is astounding 6668 fps on GeForce GTX 1080 Ti. Code is available at urlhttps://github.com/Media-Smart/vedatad.