# Future Word Contexts in Neural Network Language Models

arXiv: https://arxiv.org/abs/1708.05592 · HTML: https://mlsanity.com/paper/1708.05592/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2017.

## Built on by (most cited first, 2 of 2)

- [A Parallelizable Lattice Rescoring Strategy with Neural Language Models](https://mlsanity.com/paper/2103.05081/) (2021)
- [Back from the future: bidirectional CTC decoding using future information in speech recognition](https://mlsanity.com/paper/2110.03326/) (2021)

## Abstract (arXiv)

Recently, bidirectional recurrent network language models (bi-RNNLMs) have been shown to outperform standard, unidirectional, recurrent neural network language models (uni-RNNLMs) on a range of speech recognition tasks. This indicates that future word context information beyond the word history can be useful. However, bi-RNNLMs pose a number of challenges as they make use of the complete previous and future word context information. This impacts both training efficiency and their use within a lattice rescoring framework. In this paper these issues are addressed by proposing a novel neural network structure, succeeding word RNNLMs (su-RNNLMs). Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a finite number of succeeding, future, words. This model can be trained much more efficiently than bi-RNNLMs and can also be used for lattice rescoring. Experimental results on a meeting transcription task (AMI) show the proposed model consistently outperformed uni-RNNLMs and yield only a slight degradation compared to bi-RNNLMs in N-best rescoring. Additionally, performance improvements can be obtained using lattice rescoring and subsequent confusion network decoding.