mlsanityData to 7 Oct 2026

Pushing the Limits of Non-Autoregressive Speech Recognition

arXiv 2104.03416 ↗First posted 7 Apr 2021eess.AS cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

We combine recent advancements in end-to-end speech recognition to non-autoregressive automatic speech recognition. We push the limits of non-autoregressive state-of-the-art results for multiple datasets: LibriSpeech, Fisher+Switchboard and Wall Street Journal. Key to our recipe, we leverage CTC on giant Conformer neural network architectures with SpecAugment and wav2vec2 pre-training. We achieve 1.8%/3.6% WER on LibriSpeech test/test-other sets, 5.1%/9.8% WER on Switchboard, and 3.4% on the Wall Street Journal, all without a language model.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 7 Apr 2021.

Builds on

28 of 37 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
16 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  3. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  4. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  5. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  6. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  7. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
  8. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
  9. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 65
  10. 2019SpecAugment on Large Scale Datasetscited by 60
  11. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  12. 2016Very Deep Convolutional Networks for End-to-End Speech Recognitioncited by 34
  13. 2020Improved Mask-CTC for Non-Autoregressive End-to-End ASRcited by 32
  14. 2018Fully Convolutional Speech Recognitioncited by 19
  15. 2020An investigation of phone-based subword units for end-to-end speech recognitioncited by 12
  16. 2019Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognitioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 3; 2023: 6; 2024: 3; 2025: 1; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023End-to-End Speech Recognition: A Surveycited by 127
  2. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  3. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  4. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  5. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  6. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 27
  7. 2021Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Unitscited by 13
  8. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  9. 2023A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognitioncited by 5
  10. 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3
  11. 2023Exploring Attention Map Reuse for Efficient Transformer Neural Networkscited by 3
  12. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
6 more
  1. 2023Unsupervised Pre-Training for Vietnamese Automatic Speech Recognition in the HYKIST Projectcited by 2
  2. 2024UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Modelscited by 1
  3. 2024Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Maskcited by 1
  4. 2023Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognitioncited by 0
  5. 2023A Lexical-aware Non-autoregressive Transformer-based ASR Modelcited by 0
  6. 2026Design of the IBM Granite 5.0 TurboCTC ASR Modelcited by 0