# End-to-end Speech Recognition with Word-based RNN Language Models

arXiv: https://arxiv.org/abs/1808.02608 · HTML: https://mlsanity.com/paper/1808.02608/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2018.

## Built on by (most cited first, 23 of 23)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Transformer Language Models with LSTM-based Cross-utterance Information Representation](https://mlsanity.com/paper/2102.06474/) (2021)
- [Large-scale Language Model Rescoring on Long-form Data](https://mlsanity.com/paper/2306.08133/) (2023)
- [Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generator](https://mlsanity.com/paper/2205.09058/) (2022)
- [End2End Acoustic to Semantic Transduction](https://mlsanity.com/paper/2102.01013/) (2021)
- [Spell my name: keyword boosted speech recognition](https://mlsanity.com/paper/2110.02791/) (2021)
- [Consistent Training and Decoding For End-to-end Speech Recognition Using Lattice-free MMI](https://mlsanity.com/paper/2112.02498/) (2021)
- [Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music](https://mlsanity.com/paper/2204.03307/) (2022)
- [Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2104.00120/) (2021)
- [Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model](https://mlsanity.com/paper/2201.01995/) (2022)
- [Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model](https://mlsanity.com/paper/2303.00091/) (2023)
- [Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition](https://mlsanity.com/paper/2501.09258/) (2025)
- [Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2107.01275/) (2021)
- [Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding](https://mlsanity.com/paper/2202.05209/) (2022)
- [Capturing Multi-Resolution Context by Dilated Self-Attention](https://mlsanity.com/paper/2104.02858/) (2021)
- [Whispering Under the Eaves: Protecting User Privacy Against Commercial and LLM-powered Automatic Speech Recognition Systems](https://mlsanity.com/paper/2504.00858/) (2025)
- [Two-Stage Augmentation and Adaptive CTC Fusion for Improved Robustness of Multi-Stream End-to-End ASR](https://mlsanity.com/paper/2102.03055/) (2021)
- [End-to-End Multi-speaker ASR with Independent Vector Analysis](https://mlsanity.com/paper/2204.00218/) (2022)
- [Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulations](https://mlsanity.com/paper/2505.23846/) (2025)
- [Linguistic-Enhanced Transformer with CTC Embedding for Speech Recognition](https://mlsanity.com/paper/2210.14725/) (2022)
- [Proceedings of the 4th International Workshop on Reading Music Systems](https://mlsanity.com/paper/2211.13285/) (2022)

## Abstract (arXiv)

This paper investigates the impact of word-based RNN language models (RNN-LMs) on the performance of end-to-end automatic speech recognition (ASR). In our prior work, we have proposed a multi-level LM, in which character-based and word-based RNN-LMs are combined in hybrid CTC/attention-based ASR. Although this multi-level approach achieves significant error reduction in the Wall Street Journal (WSJ) task, two different LMs need to be trained and used for decoding, which increase the computational cost and memory usage. In this paper, we further propose a novel word-based RNN-LM, which allows us to decode with only the word-based LM, where it provides look-ahead word probabilities to predict next characters instead of the character-based LM, leading competitive accuracy with less computation compared to the multi-level LM. We demonstrate the efficacy of the word-based RNN-LMs using a larger corpus, LibriSpeech, in addition to WSJ we used in the prior work. Furthermore, we show that the proposed model achieves 5.1 %WER for WSJ Eval'92 test set when the vocabulary size is increased, which is the best WER reported for end-to-end ASR systems on this benchmark.