# How to Construct Deep Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1312.6026 · HTML: https://mlsanity.com/paper/1312.6026/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2013.

## Built on by (most cited first, 30 of 64)

- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment](https://mlsanity.com/paper/2307.03864/) (2023)
- [Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities](https://mlsanity.com/paper/2402.10835/) (2024)
- [Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2202.08756/) (2022)
- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Deep learning models for price forecasting of financial time series: A review of recent advancements: 2020-2022](https://mlsanity.com/paper/2305.04811/) (2023)
- [Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN](https://mlsanity.com/paper/2106.12901/) (2021)
- [A Novel CNN-LSTM-based Approach to Predict Urban Expansion](https://mlsanity.com/paper/2103.01695/) (2021)
- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [RF-Next: Efficient Receptive Field Search for Convolutional Neural Networks](https://mlsanity.com/paper/2206.06637/) (2022)
- [Do Transformer World Models Give Better Policy Gradients?](https://mlsanity.com/paper/2402.05290/) (2024)
- [Stacked LSTM Based Deep Recurrent Neural Network with Kalman Smoothing for Blood Glucose Prediction](https://mlsanity.com/paper/2101.06850/) (2021)
- [Multistep Electric Vehicle Charging Station Occupancy Prediction using Hybrid LSTM Neural Networks](https://mlsanity.com/paper/2106.04986/) (2021)
- [Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis](https://mlsanity.com/paper/2402.12241/) (2024)
- [Recurrent Neural Networks and Universal Approximation of Bayesian Filters](https://mlsanity.com/paper/2211.00335/) (2022)
- [Financial Table Extraction in Image Documents](https://mlsanity.com/paper/2405.05260/) (2024)
- [Hack Me If You Can: Aggregating AutoEncoders for Countering Persistent Access Threats Within Highly Imbalanced Data](https://mlsanity.com/paper/2406.19220/) (2024)
- [Pretraining Recurrent Networks without Recurrence](https://mlsanity.com/paper/2606.06479/) (2026)
- [Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluation](https://mlsanity.com/paper/2106.13393/) (2021)
- [Bayesian Non-linear Latent Variable Modeling via Random Fourier Features](https://mlsanity.com/paper/2306.08352/) (2023)
- [Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows](https://mlsanity.com/paper/2403.16995/) (2024)
- [Lower Bounds on the Expressivity of Recurrent Neural Language Models](https://mlsanity.com/paper/2405.19222/) (2024)
- [A Differential Geometry Perspective on Orthogonal Recurrent Models](https://mlsanity.com/paper/2102.09589/) (2021)
- [On the Evaluation of Sequential Machine Learning for Network Intrusion Detection](https://mlsanity.com/paper/2106.07961/) (2021)
- [Choir Transformer: Generating Polyphonic Music with Relative Attention on Transformer](https://mlsanity.com/paper/2308.02531/) (2023)
- [Classification of Alzheimers Disease with Deep Learning on Eye-tracking Data](https://mlsanity.com/paper/2309.12574/) (2023)
- [Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?](https://mlsanity.com/paper/2502.04832/) (2025)
- [When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective](https://mlsanity.com/paper/2503.11272/) (2025)
- [Supervised Visual Attention for Simultaneous Multimodal Machine Translation](https://mlsanity.com/paper/2201.09324/) (2022)

## Abstract (arXiv)

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a textitdeep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.