# Globally Normalising the Transducer for Streaming Speech Recognition

arXiv: https://arxiv.org/abs/2307.10975 · HTML: https://mlsanity.com/paper/2307.10975/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 22 of 22)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition](https://mlsanity.com/paper/2007.13802/) (2020)
- [Globally Normalized Transition-Based Neural Networks](https://mlsanity.com/paper/1603.06042/) (2016)
- [Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging](https://mlsanity.com/paper/2012.06749/) (2020)
- [Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition](https://mlsanity.com/paper/1911.12487/) (2019)
- [Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition](https://mlsanity.com/paper/2008.05750/) (2020)
- [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017)
- [Minimum Latency Training Strategies for Streaming Sequence-to-Sequence ASR](https://mlsanity.com/paper/2004.05009/) (2020)
- [A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies](https://mlsanity.com/paper/2201.05420/) (2022)
- [An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search](https://mlsanity.com/paper/1904.06834/) (2019)
- [Global Normalization for Streaming Speech Recognition in a Modular Framework](https://mlsanity.com/paper/2205.13674/) (2022)
- [Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2211.02333/) (2022)
- [Two-Pass End-to-End ASR Model Compression](https://mlsanity.com/paper/2201.02741/) (2022)

## Abstract (arXiv)

The Transducer (e.g. RNN-Transducer or Conformer-Transducer) generates an output label sequence as it traverses the input sequence. It is straightforward to use in streaming mode, where it generates partial hypotheses before the complete input has been seen. This makes it popular in speech recognition. However, in streaming mode the Transducer has a mathematical flaw which, simply put, restricts the model's ability to change its mind. The fix is to replace local normalisation (e.g. a softmax) with global normalisation, but then the loss function becomes impossible to evaluate exactly. A recent paper proposes to solve this by approximating the model, severely degrading performance. Instead, this paper proposes to approximate the loss function, allowing global normalisation to apply to a state-of-the-art streaming model. Global normalisation reduces its word error rate by 9-11% relative, closing almost half the gap between streaming and lookahead mode.