# Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling

arXiv: https://arxiv.org/abs/2010.06030 · HTML: https://mlsanity.com/paper/2010.06030/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Shrinking Bigfoot: Reducing wav2vec 2.0 footprint](https://mlsanity.com/paper/2103.15760/) (2021)
- [Streaming parallel transducer beam search with fast-slow cascaded encoders](https://mlsanity.com/paper/2203.15773/) (2022)
- [Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR](https://mlsanity.com/paper/2203.15206/) (2022)
- [SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution](https://mlsanity.com/paper/2211.11419/) (2022)
- [FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition](https://mlsanity.com/paper/2210.17079/) (2022)

## Abstract (arXiv)

Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-context ASR waits for the completion of a full speech utterance before emitting completed hypotheses. In this work, we propose a unified framework, Dual-mode ASR, to train a single end-to-end ASR model with shared weights for both streaming and full-context speech recognition. We show that the latency and accuracy of streaming ASR significantly benefit from weight sharing and joint training of full-context ASR, especially with inplace knowledge distillation during the training. The Dual-mode ASR framework can be applied to recent state-of-the-art convolution-based and transformer-based ASR networks. We present extensive experiments with two state-of-the-art ASR networks, ContextNet and Conformer, on two datasets, a widely used public dataset LibriSpeech and a large-scale dataset MultiDomain. Experiments and ablation studies demonstrate that Dual-mode ASR not only simplifies the workflow of training and deploying streaming and full-context ASR models, but also significantly improves both emission latency and recognition accuracy of streaming ASR. With Dual-mode ASR, we achieve new state-of-the-art streaming ASR results on both LibriSpeech and MultiDomain in terms of accuracy and latency.