# Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models arXiv: https://arxiv.org/abs/2107.09428 ยท HTML: https://mlsanity.com/paper/2107.09428/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 35 references that are in the map. - First posted on arXiv 20 Jul 2021. ## Builds on (most cited first, 28 of 28) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019) - [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017) - [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017) - [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019) - [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020) - [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017) - [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020) - [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020) - [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018) - [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014) - [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020) - [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020) - [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017) - [A comparison of end-to-end models for long-form speech recognition](https://mlsanity.com/paper/1911.02242/) (2019) - [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020) - [Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment](https://mlsanity.com/paper/2010.14233/) (2020) - [Improved Mask-CTC for Non-Autoregressive End-to-End ASR](https://mlsanity.com/paper/2010.13270/) (2020) - [Transformer-based Online CTC/attention End-to-End Speech Recognition Architecture](https://mlsanity.com/paper/2001.08290/) (2020) - [Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition](https://mlsanity.com/paper/2005.07903/) (2020) - [Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition](https://mlsanity.com/paper/2010.03192/) (2020) - [CASS-NAT: CTC Alignment-based Single Step Non-autoregressive Transformer for Speech Recognition](https://mlsanity.com/paper/2010.14725/) (2020) - [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019) - [Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder](https://mlsanity.com/paper/2010.13047/) (2020) - [Enhancing Monotonic Multihead Attention for Streaming ASR](https://mlsanity.com/paper/2005.09394/) (2020) - [Insertion-Based Modeling for End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2005.13211/) (2020) ## Built on by (most cited first, 11 of 11) - [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022) - [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023) - [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022) - [Deliberation of Streaming RNN-Transducer by Non-autoregressive Decoding](https://mlsanity.com/paper/2112.11442/) (2021) - [BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder](https://mlsanity.com/paper/2211.00792/) (2022) - [Semi-Autoregressive Streaming ASR With Label Context](https://mlsanity.com/paper/2309.10926/) (2023) - [Streaming Align-Refine for Non-autoregressive Deliberation](https://mlsanity.com/paper/2204.07556/) (2022) - [Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Mask](https://mlsanity.com/paper/2406.10034/) (2024) - [Korean Tokenization for Beam Search Rescoring in Speech Recognition](https://mlsanity.com/paper/2203.03583/) (2022) - [Streaming Speech-to-Confusion Network Speech Recognition](https://mlsanity.com/paper/2306.03778/) (2023) - [Segmental Attention Decoding With Long Form Acoustic Encodings](https://mlsanity.com/paper/2512.14652/) (2025) ## Abstract (arXiv) Non-autoregressive (NAR) modeling has gained more and more attention in speech processing. With recent state-of-the-art attention-based automatic speech recognition (ASR) structure, NAR can realize promising real-time factor (RTF) improvement with only small degradation of accuracy compared to the autoregressive (AR) models. However, the recognition inference needs to wait for the completion of a full speech utterance, which limits their applications on low latency scenarios. To address this issue, we propose a novel end-to-end streaming NAR speech recognition system by combining blockwise-attention and connectionist temporal classification with mask-predict (Mask-CTC) NAR. During inference, the input audio is separated into small blocks and then processed in a blockwise streaming way. To address the insertion and deletion error at the edge of the output of each block, we apply an overlapping decoding strategy with a dynamic mapping trick that can produce more coherent sentences. Experimental results show that the proposed method improves online ASR recognition in low latency conditions compared to vanilla Mask-CTC. Moreover, it can achieve a much faster inference speed compared to the AR attention-based models. All of our codes will be publicly available at https://github.com/espnet/espnet.