mlsanityData to 7 Oct 2026

End-to-End Speech Recognition: A Survey

arXiv 2303.03329 ↗First posted 3 Mar 2023eess.AS cs.CL cs.SDcited by 127

Not on the map. Why?

Abstract (arXiv)

In the last decade of automatic speech recognition (ASR) research, the introduction of deep learning brought considerable reductions in word error rate of more than 50% relative, compared to modeling without deep learning. In the wake of this transition, a number of all-neural ASR architectures were introduced. These so-called end-to-end (E2E) models provide highly integrated, completely neural ASR models, which rely strongly on general machine learning knowledge, learn more consistently from data, while depending less on ASR domain-specific experience. The success and enthusiastic adoption of deep learning accompanied by more generic model architectures lead to E2E models now becoming the prominent ASR approach. The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements, and to discuss their properties and their relation to the classical hidden Markov model (HMM) based ASR architecture. All relevant aspects of E2E ASR are covered in this work: modeling, training, decoding, and external language model integration, accompanied by discussions of performance and deployment opportunities, as well as an outlook into potential future developments.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 195 of its 350 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

195 of 350 references.

References · 195

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  8. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2016Language Modeling with Gated Convolutional Networkscited by 688
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  13. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  14. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  15. 2015Listen, Attend and Spellcited by 542
  16. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  17. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  18. 2015Attention-Based Models for Speech Recognitioncited by 400
  19. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  20. 2017Population Based Training of Neural Networkscited by 351
  21. 2017Six Challenges for Neural Machine Translationcited by 334
  22. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  23. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  24. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  25. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  26. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  27. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  28. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  29. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  30. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  31. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  32. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  33. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  34. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  35. 2018Deep context: end-to-end contextual speech recognitioncited by 136
  36. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  37. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  38. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120

50 most cited of 195 All 195 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 42; 2025: 46; 2026: 21.

127 papers.

Sort:

By area:

  1. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  2. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  3. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 56
  4. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  5. 2024Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Searchcited by 19
  6. 2024Exploring the Capability of Mamba in Speech Applicationscited by 19
  7. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  8. 2024Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speechcited by 15
  9. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  10. 2023Phoneme-aware Encoding for Prefix-tree-based Contextual ASRcited by 13
  11. 2023One model to rule them all ? Towards End-to-End Joint Speaker Diarization and Speech Recognitioncited by 12
  12. 2024On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Modelscited by 10
38 more
  1. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  2. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  3. 2024Label-Looping: Highly Efficient Decoding for Transducerscited by 9
  4. 2024Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Losscited by 9
  5. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  6. 2023BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognitioncited by 8
  7. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  8. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  9. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  10. 2024Brain-inspired Artificial Intelligence: A Comprehensive Reviewcited by 8
  11. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  12. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  13. 2024Improving child speech recognition with augmented child-like speechcited by 7
  14. 2024Innovative Speech-Based Deep Learning Approaches for Parkinson's Disease Classification: A Systematic Reviewcited by 7
  15. 2025In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varietiescited by 7
  16. 2025Measuring the Effect of Transcription Noise on Downstream Language Understanding Taskscited by 6
  17. 2025Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translationcited by 6
  18. 2025Toward Advancing License Plate Super-Resolution in Real-World Scenarios: A Dataset and Benchmarkcited by 6
  19. 2023Semi-Autoregressive Streaming ASR With Label Contextcited by 5
  20. 2024Breaking Down Power Barriers in On-Device Streaming ASR: Insights and Solutionscited by 5
  21. 2024Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Modelscited by 5
  22. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Expertscited by 5
  23. 2025Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognitioncited by 5
  24. 2025GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systemscited by 5
  25. 2023TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Modelscited by 4
  26. 2023Pseudo-Labeling for Domain-Agnostic Bangla Automatic Speech Recognitioncited by 4
  27. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  28. 2024Joint Beam Search Integrating CTC, Attention, and Transducer Decoderscited by 4
  29. 2024To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillationcited by 4
  30. 2025Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audiocited by 4
  31. 2025Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Reviewcited by 4
  32. 2025TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Treecited by 4
  33. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  34. 2023Diffusion-based Signal Refiner for Speech Enhancement and Separationcited by 3
  35. 2023DistriBlock: Identifying adversarial audio samples by leveraging characteristics of the output distributioncited by 3
  36. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  37. 2024A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligencecited by 3
  38. 2024Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillationcited by 3

50 most cited of 127 All 127 (JSON)