mlsanityData to 7 Oct 2026

A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

arXiv 1504.00941 ↗First posted 3 Apr 2015cs.NE cs.LGcited by 130

Not on the map. Why?

Abstract (arXiv)

Learning long term dependencies in recurrent networks is difficult due to vanishing and exploding gradients. To overcome this difficulty, researchers have developed sophisticated optimization techniques and network architectures. In this paper, we propose a simpler solution that use recurrent neural networks composed of rectified linear units. Key to our solution is the use of the identity matrix or its scaled version to initialize the recurrent weight matrix. We find that our solution is comparable to LSTM on our four benchmarks: two toy problems involving long-range temporal structures, a large language modeling problem and a benchmark speech recognition problem.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Apr 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 24; 2023: 20; 2024: 22; 2025: 16; 2026: 16.

130 papers.

Sort:

By area:

  1. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  2. 2022Towards Large-Scale Small Object Detection: Survey and Benchmarkscited by 119
  3. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  4. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  5. 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
  6. 2021Gradients are Not All You Needcited by 82
  7. 2024TKAN: Temporal Kolmogorov-Arnold Networkscited by 70
  8. 2024A Survey on Kolmogorov-Arnold Networkcited by 67
  9. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  10. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  11. 2021Noisy Recurrent Neural Networkscited by 42
  12. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
38 more
  1. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26
  2. 2024Universal Neural Functionalscited by 25
  3. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  4. 2021Parallelizing Legendre Memory Unit Trainingcited by 22
  5. 2021Orthogonal Graph Neural Networkscited by 19
  6. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  7. 2024SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Modelscited by 19
  8. 2021Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNNcited by 16
  9. 2022Towards a General Purpose CNN for Long Range Dependencies in NDcited by 16
  10. 2023Traveling Waves Encode the Recent Past and Enhance Sequence Learningcited by 16
  11. 2022Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networkscited by 13
  12. 2022Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomescited by 11
  13. 2024LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Unitscited by 11
  14. 2024Mamba-Spike: Enhancing the Mamba Architecture with a Spiking Front-End for Efficient Temporal Data Processingcited by 11
  15. 2021Working Memory Connections for LSTMcited by 10
  16. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  17. 2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
  18. 2024E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learningcited by 10
  19. 2024SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learningcited by 9
  20. 2022Path Development Network with Finite-dimensional Lie Group Representationcited by 8
  21. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  22. 2024Unitary convolutions for learning on graphs and groupscited by 8
  23. 2021Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processorscited by 7
  24. 2021Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decaycited by 7
  25. 2021RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networkscited by 7
  26. 2021CAN3D: Fast 3D Medical Image Segmentation via Compact Context Aggregationcited by 7
  27. 2022Exploring the Long-Term Generalization of Counting Behavior in RNNscited by 7
  28. 2023SutraNets: Sub-series Autoregressive Networks for Long-Sequence, Probabilistic Forecastingcited by 7
  29. 2026M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modelingcited by 7
  30. 2022A Natural Bias for Language Generation Modelscited by 6
  31. 2023SMPConv: Self-moving Point Representations for Continuous Convolutioncited by 6
  32. 2024Deep State Space Recurrent Neural Networks for Time Series Forecastingcited by 6
  33. 2022Semantic-based Data Augmentation for Math Word Problemscited by 5
  34. 2022Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNscited by 5
  35. 2022Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networkscited by 5
  36. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  37. 2023Stability-Informed Initialization of Neural Ordinary Differential Equationscited by 5
  38. 2025IDInit: A Universal and Stable Initialization Method for Neural Network Trainingcited by 5

50 most cited of 130 All 130 (JSON)