# Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducers

arXiv: https://arxiv.org/abs/2305.04159 · HTML: https://mlsanity.com/paper/2305.04159/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2023.

## Built on by (most cited first, 6 of 6)

- [OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification](https://mlsanity.com/paper/2402.12654/) (2024)
- [Towards Online Continuous Sign Language Recognition and Translation](https://mlsanity.com/paper/2401.05336/) (2024)
- [Mamba for Streaming ASR Combined with Unimodal Aggregation](https://mlsanity.com/paper/2410.00070/) (2024)
- [Improving RNN-Transducers with Acoustic LookAhead](https://mlsanity.com/paper/2307.05006/) (2023)
- [Speculative Speech Recognition by Audio-Prefixed Low-Rank Adaptation of Language Models](https://mlsanity.com/paper/2407.04641/) (2024)
- [Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems](https://mlsanity.com/paper/2409.19990/) (2024)

## Abstract (arXiv)

Streaming speech recognition architectures are employed for low-latency, real-time applications. Such architectures are often characterized by their causality. Causal architectures emit tokens at each frame, relying only on current and past signal, while non-causal models are exposed to a window of future frames at each step to increase predictive accuracy. This dichotomy amounts to a trade-off for real-time Automatic Speech Recognition (ASR) system design: profit from the low-latency benefit of strictly-causal architectures while accepting predictive performance limitations, or realize the modeling benefits of future-context models accompanied by their higher latency penalty. In this work, we relax the constraints of this choice and present the Adaptive Non-Causal Attention Transducer (ANCAT). Our architecture is non-causal in the traditional sense, but executes in a low-latency, streaming manner by dynamically choosing when to rely on future context and to what degree within the audio stream. The resulting mechanism, when coupled with our novel regularization algorithms, delivers comparable accuracy to non-causal configurations while improving significantly upon latency, closing the gap with their causal counterparts. We showcase our design experimentally by reporting comparative ASR task results with measures of accuracy and latency on both publicly accessible and production-scale, voice-assistant datasets.