# Towards Good Practices for Very Deep Two-Stream ConvNets

arXiv: https://arxiv.org/abs/1507.02159 · HTML: https://mlsanity.com/paper/1507.02159/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2015.

## Built on by (most cited first, 25 of 25)

- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [Temporal Context Aggregation Network for Temporal Action Proposal Refinement](https://mlsanity.com/paper/2103.13141/) (2021)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Learning from Temporal Gradient for Semi-supervised Action Recognition](https://mlsanity.com/paper/2111.13241/) (2021)
- [RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussians](https://mlsanity.com/paper/2406.11836/) (2024)
- [Hierarchical Compositional Representations for Few-shot Action Recognition](https://mlsanity.com/paper/2208.09424/) (2022)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [A 3D CNN Network with BERT For Automatic COVID-19 Diagnosis From CT-Scan Images](https://mlsanity.com/paper/2106.14403/) (2021)
- [CyclingNet: Detecting cycling near misses from video streams in complex urban scenes with deep learning](https://mlsanity.com/paper/2102.00565/) (2021)
- [Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction](https://mlsanity.com/paper/2107.05140/) (2021)
- [RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model](https://mlsanity.com/paper/2407.06162/) (2024)
- [Searching for Two-Stream Models in Multivariate Space for Video Recognition](https://mlsanity.com/paper/2108.12957/) (2021)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [PLSM: A Parallelized Liquid State Machine for Unintentional Action Detection](https://mlsanity.com/paper/2105.09909/) (2021)
- [Robotic Detection of a Human-Comprehensible Gestural Language for Underwater Multi-Human-Robot Collaboration](https://mlsanity.com/paper/2207.05331/) (2022)
- [Transferable Knowledge-Based Multi-Granularity Aggregation Network for Temporal Action Localization: Submission to ActivityNet Challenge 2021](https://mlsanity.com/paper/2107.12618/) (2021)
- [Temporal Action Localization Using Gated Recurrent Units](https://mlsanity.com/paper/2108.03375/) (2021)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [DroneAttention: Sparse Weighted Temporal Attention for Drone-Camera Based Activity Recognition](https://mlsanity.com/paper/2212.03384/) (2022)
- [Deep set conditioned latent representations for action recognition](https://mlsanity.com/paper/2212.11030/) (2022)
- [Unlimited Knowledge Distillation for Action Recognition in the Dark](https://mlsanity.com/paper/2308.09327/) (2023)
- [SWTF: Sparse Weighted Temporal Fusion for Drone-Based Activity Recognition](https://mlsanity.com/paper/2211.05531/) (2022)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)
- [Semantically-Aware Diver Activity Recognition Framework for Effective Underwater Multi-Human-Robot Collaboration](https://mlsanity.com/paper/2606.12374/) (2026)

## Abstract (arXiv)

Deep convolutional networks have achieved great success for object recognition in still images. However, for action recognition in videos, the improvement of deep convolutional networks is not so evident. We argue that there are two reasons that could probably explain this result. First the current network architectures (e.g. Two-stream ConvNets) are relatively shallow compared with those very deep models in image domain (e.g. VGGNet, GoogLeNet), and therefore their modeling capacity is constrained by their depth. Second, probably more importantly, the training dataset of action recognition is extremely small compared with the ImageNet dataset, and thus it will be easy to over-fit on the training dataset. To address these issues, this report presents very deep two-stream ConvNets for action recognition, by adapting recent very deep architectures into video domain. However, this extension is not easy as the size of action recognition is quite small. We design several good practices for the training of very deep two-stream ConvNets, namely (i) pre-training for both spatial and temporal nets, (ii) smaller learning rates, (iii) more data augmentation techniques, (iv) high drop out ratio. Meanwhile, we extend the Caffe toolbox into Multi-GPU implementation with high computational efficiency and low memory consumption. We verify the performance of very deep two-stream ConvNets on the dataset of UCF101 and it achieves the recognition accuracy of 91.4%.