# Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition

arXiv: https://arxiv.org/abs/1701.03360 · HTML: https://mlsanity.com/paper/1701.03360/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2017.

## Built on by (most cited first, 19 of 19)

- [ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models](https://mlsanity.com/paper/2106.10121/) (2021)
- [Memory-efficient Speech Recognition on Smart Devices](https://mlsanity.com/paper/2102.11531/) (2021)
- [ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees](https://mlsanity.com/paper/2111.05496/) (2021)
- [ESCORT: Ethereum Smart COntRacTs Vulnerability Detection using Deep Neural Network and Transfer Learning](https://mlsanity.com/paper/2103.12607/) (2021)
- [PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction](https://mlsanity.com/paper/2312.00839/) (2023)
- [MambaSL: Exploring Single-Layer Mamba for Time Series Classification](https://mlsanity.com/paper/2604.15174/) (2026)
- [Persistence Initialization: A novel adaptation of the Transformer architecture for Time Series Forecasting](https://mlsanity.com/paper/2208.14236/) (2022)
- [Fusion Flow-enhanced Graph Pooling Residual Networks for Unmanned Aerial Vehicles Surveillance in Day and Night Dual Visions](https://mlsanity.com/paper/2407.12647/) (2024)
- [Gates Are Not What You Need in RNNs](https://mlsanity.com/paper/2108.00527/) (2021)
- [Real-time Forecasting of Time Series in Financial Markets Using Sequentially Trained Many-to-one LSTMs](https://mlsanity.com/paper/2205.04678/) (2022)
- [Temporal Spatial Decomposition and Fusion Network for Time Series Forecasting](https://mlsanity.com/paper/2210.03122/) (2022)
- [How to choose "Good" Samples for Text Data Augmentation](https://mlsanity.com/paper/2302.00894/) (2023)
- [Impact of translation on biomedical information extraction from real-life clinical notes](https://mlsanity.com/paper/2306.02042/) (2023)
- [Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2606.20411/) (2026)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [Parameter Estimation for the SEIR Model Using Recurrent Nets](https://mlsanity.com/paper/2105.14524/) (2021)
- [Hysteretic Behavior Simulation Based on Pyramid Neural Network:Principle, Network Architecture, Case Study and Explanation](https://mlsanity.com/paper/2206.03990/) (2022)
- [Semi-supervised 3D Video Information Retrieval with Deep Neural Network and Bi-directional Dynamic-time Warping Algorithm](https://mlsanity.com/paper/2309.01063/) (2023)
- [American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM](https://mlsanity.com/paper/2409.10874/) (2024)

## Abstract (arXiv)

In this paper, a novel architecture for a deep recurrent neural network, residual LSTM is introduced. A plain LSTM has an internal memory cell that can learn long term dependencies of sequential data. It also provides a temporal shortcut path to avoid vanishing or exploding gradients in the temporal domain. The residual LSTM provides an additional spatial shortcut path from lower layers for efficient training of deep networks with multiple LSTM layers. Compared with the previous work, highway LSTM, residual LSTM separates a spatial shortcut path with temporal one by using output layers, which can help to avoid a conflict between spatial and temporal-domain gradient flows. Furthermore, residual LSTM reuses the output projection matrix and the output gate of LSTM to control the spatial information flow instead of additional gate networks, which effectively reduces more than 10% of network parameters. An experiment for distant speech recognition on the AMI SDM corpus shows that 10-layer plain and highway LSTM networks presented 13.7% and 6.2% increase in WER over 3-layer aselines, respectively. On the contrary, 10-layer residual LSTM networks provided the lowest WER 41.0%, which corresponds to 3.3% and 2.8% WER reduction over plain and highway LSTM networks, respectively.