mlsanityData to 7 Oct 2026

How to Construct Deep Recurrent Neural Networks

arXiv 1312.6026 ↗First posted 20 Dec 2013cs.NE cs.LG stat.MLcited by 64

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore different ways to extend a recurrent neural network (RNN) to a textitdeep RNN. We start by arguing that the concept of depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, however, we find three points of an RNN which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. Based on this observation, we propose two novel architectures of a deep RNN which are orthogonal to an earlier attempt of stacking multiple recurrent layers to build a deep RNN (Schmidhuber, 1992; El Hihi and Bengio, 1996). We provide an alternative interpretation of these deep RNNs using a novel framework based on neural operators. The proposed deep RNNs are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNNs benefit from the depth and outperform the conventional, shallow RNNs.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 15; 2023: 9; 2024: 14; 2025: 7; 2026: 5.

64 papers.

Sort:

By area:

  1. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  2. 2023Deep Learning for Time Series Classification and Extrinsic Regression: A Current Surveycited by 68
  3. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  4. 2024Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilitiescited by 48
  5. 2022Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecastingcited by 27
  6. 2023A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifoldcited by 21
  7. 2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
  8. 2023Deep learning models for price forecasting of financial time series: A review of recent advancements: 2020-2022cited by 16
  9. 2021A Novel CNN-LSTM-based Approach to Predict Urban Expansioncited by 13
  10. 2024Deep Learning for Satellite Image Time Series Analysis: A Reviewcited by 13
  11. 2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
  12. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
38 more
  1. 2021Stacked LSTM Based Deep Recurrent Neural Network with Kalman Smoothing for Blood Glucose Predictioncited by 8
  2. 2021Multistep Electric Vehicle Charging Station Occupancy Prediction using Hybrid LSTM Neural Networkscited by 8
  3. 2024Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysiscited by 7
  4. 2022Recurrent Neural Networks and Universal Approximation of Bayesian Filterscited by 5
  5. 2024Financial Table Extraction in Image Documentscited by 5
  6. 2024Hack Me If You Can: Aggregating AutoEncoders for Countering Persistent Access Threats Within Highly Imbalanced Datacited by 5
  7. 2026Pretraining Recurrent Networks without Recurrencecited by 5
  8. 2021Interpreting Depression From Question-wise Long-term Video Recording of SDS Evaluationcited by 4
  9. 2023Bayesian Non-linear Latent Variable Modeling via Random Fourier Featurescited by 4
  10. 2024Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flowscited by 4
  11. 2024Lower Bounds on the Expressivity of Recurrent Neural Language Modelscited by 4
  12. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3
  13. 2021On the Evaluation of Sequential Machine Learning for Network Intrusion Detectioncited by 3
  14. 2023Classification of Alzheimers Disease with Deep Learning on Eye-tracking Datacited by 3
  15. 2025Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?cited by 3
  16. 2022Supervised Visual Attention for Simultaneous Multimodal Machine Translationcited by 2
  17. 2022NeuroView-RNN: It's About Timecited by 2
  18. 2022Towards Theoretical Analysis of Transformation Complexity of ReLU DNNscited by 2
  19. 2022Machine and Deep Learning for IoT Security and Privacy: Applications, Challenges, and Future Directionscited by 2
  20. 2022Re-visiting Reservoir Computing architectures optimized by Evolutionary Algorithmscited by 2
  21. 2023Choir Transformer: Generating Polyphonic Music with Relative Attention on Transformercited by 2
  22. 2024Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networkscited by 2
  23. 2024EchoSpike Predictive Plasticity: An Online Local Learning Rule for Spiking Neural Networkscited by 2
  24. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2
  25. 2026Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalizationcited by 2
  26. 2021SensPick: Sense Picking for Word Sense Disambiguationcited by 1
  27. 2021Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Datacited by 1
  28. 2021Spatio-Temporal Data Mining for Aviation Delay Predictioncited by 1
  29. 2022DeepBayes -- an estimator for parameter estimation in stochastic nonlinear dynamical modelscited by 1
  30. 2022Biometric Signature Verification Using Recurrent Neural Networkscited by 1
  31. 2022Universal Time-Uniform Trajectory Approximation for Random Dynamical Systems with Recurrent Neural Networkscited by 1
  32. 2024Machine Learning in Short-Reach Optical Systems: A Comprehensive Surveycited by 1
  33. 2025COMPOL: A Unified Neural Operator Framework for Scalable Multi-Physics Simulationscited by 1
  34. 2021Advances in Electron Microscopy with Deep Learningcited by 0
  35. 2021Dynamical System Parameter Identification using Deep Recurrent Cell Networkscited by 0
  36. 2021Energy time series forecasting-Analytical and empirical assessment of conventional and machine learning modelscited by 0
  37. 2022On Principal Curve-Based Classifiers and Similarity-Based Selective Sampling in Time-Seriescited by 0
  38. 2022Learning Individual Interactions from Population Dynamics with Discrete-Event Simulation Modelcited by 0

50 most cited of 64 All 64 (JSON)