mlsanityData to 7 Oct 2026

Recurrent Highway Networks

arXiv 1607.03474 ↗First posted 12 Jul 2016cs.LG cs.CL cs.NEcited by 36

Not on the map. Why?

Abstract (arXiv)

Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depths larger than one. Several language modeling experiments demonstrate that the proposed architecture results in powerful and efficient models. On the Penn Treebank corpus, solely increasing the transition depth from 1 to 10 improves word-level perplexity from 90.6 to 65.4 using the same number of parameters. On the larger Wikipedia datasets for character prediction (text8 and enwik8), RHNs outperform all previous results and achieve an entropy of 1.27 bits per character.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 7; 2024: 6; 2025: 4; 2026: 3.

36 papers.

Sort:

By area:

  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  2. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  3. 2021Selfish Sparse RNN Trainingcited by 26
  4. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  5. 2025Pre-training under infinite computecited by 24
  6. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  7. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  8. 2021Multi-Task Attentive Residual Networks for Argument Miningcited by 6
  9. 2021Doping: A technique for efficient compression of LSTM models using sparse structured additive matricescited by 5
  10. 2022Local Byte Fusion for Neural Machine Translationcited by 5
  11. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  12. 2022Entangled Residual Mappingscited by 4
24 more
  1. 2022Quiver neural networkscited by 4
  2. 2023A Critical Re-evaluation of Benchmark Datasets for (Deep) Learning-Based Matching Algorithmscited by 4
  3. 2021Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modelingcited by 3
  4. 2022Unfolding AIS transmission behavior for vessel movement modeling on noisy data leveraging machine learningcited by 3
  5. 2022State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictionscited by 2
  6. 2022Dependency-based Mixture Language Modelscited by 1
  7. 2022Rethinking skip connection model as a learnable Markov chaincited by 1
  8. 2023DDNAS: Discretized Differentiable Neural Architecture Search for Text Classificationcited by 1
  9. 2024Uncertainty-Aware Deep Attention Recurrent Neural Network for Heterogeneous Time Series Imputationcited by 1
  10. 2024Deep Companion Learning: Enhancing Generalization Through Historical Consistencycited by 1
  11. 2025Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrumcited by 1
  12. 2026Temporal Recurrence Favors Fewer Layerscited by 1
  13. 2022Syntactic Inductive Biases for Deep Learning Methodscited by 0
  14. 2022An Improved Time Feedforward Connections Recurrent Neural Networkscited by 0
  15. 2023Extending Memory for Language Modellingcited by 0
  16. 2023A Quantitative Review on Language Model Efficiency Researchcited by 0
  17. 2023RigLSTM: Recurrent Independent Grid LSTM for Generalizable Sequence Learningcited by 0
  18. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  19. 2024Some variation of COBRA in sequential learning setupcited by 0
  20. 2024A Hybrid Deep Learning Framework for Stock Price Prediction Considering the Investor Sentiment of Online Forum Enhanced by Popularitycited by 0
  21. 2024AdaResNet: Enhancing Residual Networks with Dynamic Weight Adjustment for Improved Feature Integrationcited by 0
  22. 2025Sparsified State-Space Models are Efficient Highway Networkscited by 0
  23. 2026Reshape and Recur: Improving SSMs with Input Reshaping and Depth Recurrencecited by 0
  24. 2026Error Highways: Scaling Predictive Coding to Very Deep Networkscited by 0