# On Training Recurrent Networks with Truncated Backpropagation Through Time in Speech Recognition

arXiv: https://arxiv.org/abs/1807.03396 · HTML: https://mlsanity.com/paper/1807.03396/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2018.

## Built on by (most cited first, 5 of 5)

- [Accurate deep learning-based filtering for chaotic dynamics by identifying instabilities without an ensemble](https://mlsanity.com/paper/2408.04739/) (2024)
- [Effective Context in Neural Speech Models](https://mlsanity.com/paper/2505.22487/) (2025)
- [Learning active tactile perception through belief-space control](https://mlsanity.com/paper/2312.00215/) (2023)
- [Tuning the burn-in phase in training recurrent neural networks improves their performance](https://mlsanity.com/paper/2602.10911/) (2026)
- [Time-series Initialization and Conditioning for Video-agnostic Stabilization of Video Super-Resolution using Recurrent Networks](https://mlsanity.com/paper/2403.15832/) (2024)

## Abstract (arXiv)

Recurrent neural networks have been the dominant models for many speech and language processing tasks. However, we understand little about the behavior and the class of functions recurrent networks can realize. Moreover, the heuristics used during training complicate the analyses. In this paper, we study recurrent networks' ability to learn long-term dependency in the context of speech recognition. We consider two decoding approaches, online and batch decoding, and show the classes of functions to which the decoding approaches correspond. We then draw a connection between batch decoding and a popular training approach for recurrent networks, truncated backpropagation through time. Changing the decoding approach restricts the amount of past history recurrent networks can use for prediction, allowing us to analyze their ability to remember. Empirically, we utilize long-term dependency in subphonetic states, phonemes, and words, and show how the design decisions, such as the decoding approach, lookahead, context frames, and consecutive prediction, characterize the behavior of recurrent networks. Finally, we draw a connection between Markov processes and vanishing gradients. These results have implications for studying the long-term dependency in speech data and how these properties are learned by recurrent networks.