# Improving RNN-Transducers with Acoustic LookAhead

arXiv: https://arxiv.org/abs/2307.05006 · HTML: https://mlsanity.com/paper/2307.05006/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Factorized Neural Transducer for Efficient Language Model Adaptation](https://mlsanity.com/paper/2110.01500/) (2021)
- [RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions](https://mlsanity.com/paper/2005.03271/) (2020)
- [Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network](https://mlsanity.com/paper/2104.11127/) (2021)
- [Improving the fusion of acoustic and text representations in RNN-T](https://mlsanity.com/paper/2201.10240/) (2022)
- [Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion](https://mlsanity.com/paper/2012.00133/) (2020)
- [Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducers](https://mlsanity.com/paper/2305.04159/) (2023)
- [Adaptive Discounting of Implicit Language Models in RNN-Transducers](https://mlsanity.com/paper/2203.02317/) (2022)
- [VQ-T: RNN Transducers using Vector-Quantized Prediction Network States](https://mlsanity.com/paper/2208.01818/) (2022)

## Built on by (most cited first, 4 of 4)

- [LINOCS: Lookahead Inference of Networked Operators for Continuous Stability](https://mlsanity.com/paper/2404.18267/) (2024)
- [Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding](https://mlsanity.com/paper/2601.15339/) (2026)
- [CodeVaani: A Multilingual, Voice-Based Code Learning Assistant](https://mlsanity.com/paper/2511.20654/) (2025)
- [The Anatomy of an ASR Hallucination](https://mlsanity.com/paper/2609.04404/) (2026)

## Abstract (arXiv)

RNN-Transducers (RNN-Ts) have gained widespread acceptance as an end-to-end model for speech to text conversion because of their high accuracy and streaming capabilities. A typical RNN-T independently encodes the input audio and the text context, and combines the two encodings by a thin joint network. While this architecture provides SOTA streaming accuracy, it also makes the model vulnerable to strong LM biasing which manifests as multi-step hallucination of text without acoustic evidence. In this paper we propose LookAhead that makes text representations more acoustically grounded by looking ahead into the future within the audio input. This technique yields a significant 5%-20% relative reduction in word error rate on both in-domain and out-of-domain evaluation sets.