# Deep RNN Framework for Visual Sequential Applications

arXiv: https://arxiv.org/abs/1811.09961 · HTML: https://mlsanity.com/paper/1811.09961/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2018.

## Built on by (most cited first, 9 of 9)

- [HybrIK-X: Hybrid Analytical-Neural Inverse Kinematics for Whole-body Mesh Recovery](https://mlsanity.com/paper/2304.05690/) (2023)
- [VIL-100: A New Dataset and A Baseline Model for Video Instance Lane Detection](https://mlsanity.com/paper/2108.08482/) (2021)
- [PGT: A Progressive Method for Training Models on Long Videos](https://mlsanity.com/paper/2103.11313/) (2021)
- [Physics-informed generative neural network: an application to troposphere temperature prediction](https://mlsanity.com/paper/2107.06991/) (2021)
- [An Efficient Framework for Few-shot Skeleton-based Temporal Action Segmentation](https://mlsanity.com/paper/2207.09925/) (2022)
- [Lyapunov-Guided Representation of Recurrent Neural Network Performance](https://mlsanity.com/paper/2204.04876/) (2022)
- [MOSformer: Momentum encoder-based inter-slice fusion transformer for medical image segmentation](https://mlsanity.com/paper/2401.11856/) (2024)
- [Reservoir Network with Structural Plasticity for Human Activity Recognition](https://mlsanity.com/paper/2503.00393/) (2025)
- [Applied Machine Learning Methods with Long-Short Term Memory Based Recurrent Neural Networks for Multivariate Temperature Prediction](https://mlsanity.com/paper/2503.06278/) (2025)

## Abstract (arXiv)

Extracting temporal and representation features efficiently plays a pivotal role in understanding visual sequence information. To deal with this, we propose a new recurrent neural framework that can be stacked deep effectively. There are mainly two novel designs in our deep RNN framework: one is a new RNN module called Context Bridge Module (CBM) which splits the information flowing along the sequence (temporal direction) and along depth (spatial representation direction), making it easier to train when building deep by balancing these two directions; the other is the Overlap Coherence Training Scheme that reduces the training complexity for long visual sequential tasks on account of the limitation of computing resources. We provide empirical evidence to show that our deep RNN framework is easy to optimize and can gain accuracy from the increased depth on several visual sequence problems. On these tasks, we evaluate our deep RNN framework with 15 layers, 7* than conventional RNN networks, but it is still easy to train. Our deep framework achieves more than 11% relative improvements over shallow RNN models on Kinetics, UCF-101, and HMDB-51 for video classification. For auxiliary annotation, after replacing the shallow RNN part of Polygon-RNN with our 15-layer deep CBM, the performance improves by 14.7%. For video future prediction, our deep RNN improves the state-of-the-art shallow model's performance by 2.4% on PSNR and SSIM. The code and trained models are published accompanied by this paper: https://github.com/BoPang1996/Deep-RNN-Framework.