mlsanityData to 7 Oct 2026

Recent Advances in End-to-End Automatic Speech Recognition

arXiv 2111.01690 ↗First posted 2 Nov 2021eess.AS cs.AI cs.CLcited by 138

Not on the map. Why?

Abstract (arXiv)

Recently, the speech community is seeing a significant trend of moving from deep neural network based hybrid modeling to end-to-end (E2E) modeling for automatic speech recognition (ASR). While E2E models achieve the state-of-the-art results in most benchmarks in terms of ASR accuracy, hybrid models are still used in a large proportion of commercial ASR systems at the current time. There are lots of practical factors that affect the production model deployment decision. Traditional hybrid models, being optimized for production for decades, are usually good at these factors. Without providing excellent solutions to all these factors, it is hard for E2E models to be widely commercialized. In this paper, we will overview the recent advances in E2E models, focusing on technologies addressing those challenges from the industry's perspective.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 247 of its 353 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

247 of 353 references.

References · 247

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Layer Normalizationcited by 5,579
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  12. 2015Listen, Attend and Spellcited by 542
38 more
  1. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  2. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  3. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  5. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  6. 2018Deep Audio-Visual Speech Recognitioncited by 318
  7. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  8. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  9. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  10. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  11. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  12. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  13. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  14. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  15. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  16. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  17. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  18. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  19. 2018Deep context: end-to-end contextual speech recognitioncited by 136
  20. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  21. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  22. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  23. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  24. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  25. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  26. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  27. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  28. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  29. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  30. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
  31. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  32. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  33. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  34. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  35. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  36. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  37. 2020Contextual RNN-T For Open Domain ASRcited by 85
  38. 2017Monotonic Chunkwise Attentioncited by 82

50 most cited of 247 All 247 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 50; 2024: 35; 2025: 24; 2026: 4.

138 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  3. 2023End-to-End Speech Recognition: A Surveycited by 127
  4. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  5. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  6. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  7. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  8. 2022WeNet 2.0: More Productive End-to-End Speech Recognition Toolkitcited by 46
  9. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  10. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  11. 2023Improving Continuous Sign Language Recognition with Cross-Lingual Signscited by 27
  12. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
38 more
  1. 2024Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Searchcited by 19
  2. 2024CTC-Assisted LLM-Based Contextual ASRcited by 18
  3. 2022Towards Contextual Spelling Correction for Customization of End-to-end Speech Recognition Systemscited by 17
  4. 2022Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languagescited by 16
  5. 2024Measuring the Accuracy of Automatic Speech Recognition Solutionscited by 15
  6. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  7. 2022Variable Attention Masking for Configurable Transformer Transducer Speech Recognitioncited by 14
  8. 2023Accelerating Transducers through Adjacent Token Mergingcited by 14
  9. 2023Multi-Scale Attention for Audio Question Answeringcited by 13
  10. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  11. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  12. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  13. 2024SoccerNet-Echoes: A Soccer Game Audio Commentary Datasetcited by 12
  14. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
  15. 2023Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Makingcited by 11
  16. 2023Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generatorcited by 11
  17. 2023CB-Conformer: Contextual biasing Conformer for biased word recognitioncited by 11
  18. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  19. 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 11
  20. 2022Boosting Cross-Domain Speech Recognition with Self-Supervisioncited by 10
  21. 2022Improving Deliberation by Text-Only and Semi-Supervised Trainingcited by 10
  22. 2022Simulating realistic speech overlaps improves multi-talker ASRcited by 10
  23. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  24. 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
  25. 2023Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentationcited by 9
  26. 2023Online Spatio-Temporal Learning with Target Projectioncited by 9
  27. 2023Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversioncited by 9
  28. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  29. 2024Cross-Speaker Encoding Network for Multi-Talker Speech Recognitioncited by 9
  30. 2024Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Losscited by 9
  31. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Modelscited by 9
  32. 2022Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognitioncited by 8
  33. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  34. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
  35. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  36. 2023ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correctioncited by 7
  37. 2024Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognitioncited by 7
  38. 2022LongFNT: Long-form Speech Recognition with Factorized Neural Transducercited by 6

50 most cited of 138 All 138 (JSON)