# Streaming Align-Refine for Non-autoregressive Deliberation

arXiv: https://arxiv.org/abs/2204.07556 · HTML: https://mlsanity.com/paper/2204.07556/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020)
- [Deliberation Model Based Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/2003.07962/) (2020)
- [Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment](https://mlsanity.com/paper/2010.14233/) (2020)
- [Improved Mask-CTC for Non-Autoregressive End-to-End ASR](https://mlsanity.com/paper/2010.13270/) (2020)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017)
- [Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models](https://mlsanity.com/paper/2201.10103/) (2022)
- [Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding](https://mlsanity.com/paper/2112.11442/) (2021)
- [Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models](https://mlsanity.com/paper/2107.09428/) (2021)
- [Toward Streaming ASR with Non-Autoregressive Insertion-based Model](https://mlsanity.com/paper/2012.10128/) (2020)

## Built on by (most cited first, 1 of 1)

- [Lego-Features: Exporting modular encoder features for streaming and deliberation ASR](https://mlsanity.com/paper/2304.00173/) (2023)

## Abstract (arXiv)

We propose a streaming non-autoregressive (non-AR) decoding algorithm to deliberate the hypothesis alignment of a streaming RNN-T model. Our algorithm facilitates a simple greedy decoding procedure, and at the same time is capable of producing the decoding result at each frame with limited right context, thus enjoying both high efficiency and low latency. These advantages are achieved by converting the offline Align-Refine algorithm to be streaming-compatible, with a novel transformer decoder architecture that performs local self-attentions for both text and audio, and a time-aligned cross-attention at each layer. Furthermore, we perform discriminative training of our model with the minimum word error rate (MWER) criterion, which has not been done in the non-AR decoding literature. Experiments on voice search datasets and Librispeech show that with reasonable right context, our streaming model performs as well as the offline counterpart, and discriminative training leads to further WER gain when the first-pass model has small capacity.