# Recurrent Highway Networks

arXiv: https://arxiv.org/abs/1607.03474 · HTML: https://mlsanity.com/paper/1607.03474/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2016.

## Built on by (most cited first, 30 of 37)

- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Unit Scaling: Out-of-the-Box Low-Precision Training](https://mlsanity.com/paper/2303.11257/) (2023)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Multi-Task Attentive Residual Networks for Argument Mining](https://mlsanity.com/paper/2102.12227/) (2021)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Local Byte Fusion for Neural Machine Translation](https://mlsanity.com/paper/2205.11490/) (2022)
- [Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians](https://mlsanity.com/paper/2312.17306/) (2023)
- [Entangled Residual Mappings](https://mlsanity.com/paper/2206.01261/) (2022)
- [Quiver neural networks](https://mlsanity.com/paper/2207.12773/) (2022)
- [A Critical Re-evaluation of Benchmark Datasets for (Deep) Learning-Based Matching Algorithms](https://mlsanity.com/paper/2307.01231/) (2023)
- [Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling](https://mlsanity.com/paper/2110.06021/) (2021)
- [Unfolding AIS transmission behavior for vessel movement modeling on noisy data leveraging machine learning](https://mlsanity.com/paper/2202.13867/) (2022)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Dependency-based Mixture Language Models](https://mlsanity.com/paper/2203.10256/) (2022)
- [Rethinking skip connection model as a learnable Markov chain](https://mlsanity.com/paper/2209.15278/) (2022)
- [DDNAS: Discretized Differentiable Neural Architecture Search for Text Classification](https://mlsanity.com/paper/2307.06005/) (2023)
- [Uncertainty-Aware Deep Attention Recurrent Neural Network for Heterogeneous Time Series Imputation](https://mlsanity.com/paper/2401.02258/) (2024)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Deep Companion Learning: Enhancing Generalization Through Historical Consistency](https://mlsanity.com/paper/2407.18821/) (2024)
- [Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum](https://mlsanity.com/paper/2506.07975/) (2025)
- [Temporal Recurrence Favors Fewer Layers](https://mlsanity.com/paper/2609.12531/) (2026)
- [Syntactic Inductive Biases for Deep Learning Methods](https://mlsanity.com/paper/2206.04806/) (2022)
- [An Improved Time Feedforward Connections Recurrent Neural Networks](https://mlsanity.com/paper/2211.02561/) (2022)
- [Extending Memory for Language Modelling](https://mlsanity.com/paper/2305.11462/) (2023)
- [A Quantitative Review on Language Model Efficiency Research](https://mlsanity.com/paper/2306.01768/) (2023)
- [RigLSTM: Recurrent Independent Grid LSTM for Generalizable Sequence Learning](https://mlsanity.com/paper/2311.02123/) (2023)

## Abstract (arXiv)

Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depths larger than one. Several language modeling experiments demonstrate that the proposed architecture results in powerful and efficient models. On the Penn Treebank corpus, solely increasing the transition depth from 1 to 10 improves word-level perplexity from 90.6 to 65.4 using the same number of parameters. On the larger Wikipedia datasets for character prediction (text8 and enwik8), RHNs outperform all previous results and achieve an entropy of 1.27 bits per character.