# A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation

arXiv: https://arxiv.org/abs/2110.05249 · HTML: https://mlsanity.com/paper/2110.05249/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 66 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions](https://mlsanity.com/paper/1910.10261/) (2019)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018)
- [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020)

## Built on by (most cited first, 28 of 28)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification](https://mlsanity.com/paper/2402.12654/) (2024)
- [Improving Massively Multilingual ASR With Auxiliary CTC Objectives](https://mlsanity.com/paper/2302.12829/) (2023)
- [A context-aware knowledge transferring strategy for CTC-based ASR](https://mlsanity.com/paper/2210.06244/) (2022)
- [Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translation](https://mlsanity.com/paper/2204.08920/) (2022)
- [BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model](https://mlsanity.com/paper/2210.16663/) (2022)
- [Conformer with dual-mode chunked attention for joint online and offline ASR](https://mlsanity.com/paper/2206.11157/) (2022)
- [CTC-based Non-autoregressive Speech Translation](https://mlsanity.com/paper/2305.17358/) (2023)
- [CIF-T: A Novel CIF-based Transducer Architecture for Automatic Speech Recognition](https://mlsanity.com/paper/2307.14132/) (2023)
- [A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition](https://mlsanity.com/paper/2304.07611/) (2023)
- [InterBiasing: Boost Unseen Word Recognition through Biasing Intermediate Predictions](https://mlsanity.com/paper/2406.14890/) (2024)
- [Improving CTC-based ASR Models with Gated Interlayer Collaboration](https://mlsanity.com/paper/2205.12462/) (2022)
- [MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognition](https://mlsanity.com/paper/2202.08456/) (2022)
- [Memory-Efficient Training of RNN-Transducer with Sampled Softmax](https://mlsanity.com/paper/2203.16868/) (2022)
- [A Token-Wise Beam Search Algorithm for RNN-T](https://mlsanity.com/paper/2302.14357/) (2023)
- [Neural Diarization with Non-autoregressive Intermediate Attractors](https://mlsanity.com/paper/2303.06806/) (2023)
- [WCTC-Biasing: Retraining-free Contextual Biasing ASR with Wildcard CTC-based Keyword Spotting and Inter-layer Biasing](https://mlsanity.com/paper/2506.01263/) (2025)
- [Unimodal Aggregation for CTC-based Speech Recognition](https://mlsanity.com/paper/2309.08150/) (2023)
- [DYNAC: Dynamic Vocabulary based Non-Autoregressive Contextualization for Speech Recognition](https://mlsanity.com/paper/2506.00422/) (2025)
- [On the Prediction Network Architecture in RNN-T for ASR](https://mlsanity.com/paper/2206.14618/) (2022)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models](https://mlsanity.com/paper/2402.08898/) (2024)
- [Alternate Intermediate Conditioning with Syllable-level and Character-level Targets for Japanese ASR](https://mlsanity.com/paper/2204.00175/) (2022)
- [InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss](https://mlsanity.com/paper/2211.00795/) (2022)
- [A Lexical-aware Non-autoregressive Transformer-based ASR Model](https://mlsanity.com/paper/2305.10839/) (2023)
- [UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognition](https://mlsanity.com/paper/2509.14653/) (2025)
- [Diffusion Language Models for Speech Recognition](https://mlsanity.com/paper/2604.14001/) (2026)

## Abstract (arXiv)

Non-autoregressive (NAR) models simultaneously generate multiple outputs in a sequence, which significantly reduces the inference speed at the cost of accuracy drop compared to autoregressive baselines. Showing great potential for real-time applications, an increasing number of NAR models have been explored in different fields to mitigate the performance gap against AR models. In this work, we conduct a comparative study of various NAR modeling methods for end-to-end automatic speech recognition (ASR). Experiments are performed in the state-of-the-art setting using ESPnet. The results on various tasks provide interesting findings for developing an understanding of NAR ASR, such as the accuracy-speed trade-off and robustness against long-form utterances. We also show that the techniques can be combined for further improvement and applied to NAR end-to-end speech translation. All the implementations are publicly available to encourage further research in NAR speech processing.