mlsanityData to 7 Oct 2026

An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition

arXiv 2110.04590 ↗First posted 9 Oct 2021cs.CL cs.SD eess.AScited by 39

Not on the map. Why?

Abstract (arXiv)

Self-supervised pretraining on speech data has achieved a lot of progress. High-fidelity representation of the speech signal is learned from a lot of untranscribed data and shows promising performance. Recently, there are several works focusing on evaluating the quality of self-supervised pretrained representations on various tasks without domain restriction, e.g. SUPERB. However, such evaluations do not provide a comprehensive comparison among many ASR benchmark corpora. In this paper, we focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR. Without any modification of the back-end model architectures or training strategy, some of the experiments with pretrained representations, e.g., WSJ, WSJ0-2mix with HuBERT, reach or outperform current state-of-the-art (SOTA) recognition performance. Moreover, we further explore more scenarios for whether the pretraining representations are effective, such as the cross-language or overlapped speech. The scripts, configuratons and the trained models have been released in ESPnet to let the community reproduce our experiments and improve them.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

32 of 51 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  7. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  8. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
20 more
  1. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  2. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  3. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  4. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  5. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  6. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  7. 2018Scaling Neural Machine Translationcited by 204
  8. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  9. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  10. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  11. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  12. 2019Self-Training for End-to-End Speech Recognitioncited by 128
  13. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  14. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  15. 2020Unsupervised pretraining transfers well across languagescited by 100
  16. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  17. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  18. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  19. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  20. 2020ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integrationcited by 22

Built on by

Cited by year

Citing papers per year: 2022: 13; 2023: 12; 2024: 12; 2025: 2.

39 papers.

Sort:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  3. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  4. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  5. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  6. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  7. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  8. 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
  9. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  10. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  11. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  12. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
27 more
  1. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  2. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translationcited by 11
  3. 2023Investigating Pre-trained Audio Encoders in the Low-Resource Conditioncited by 11
  4. 2024Useful Blunders: Can Automated Speech Recognition Errors Improve Downstream Dementia Classification?cited by 9
  5. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 8
  6. 2022FeaRLESS: Feature Refinement Loss for Ensembling Self-Supervised Learning Features in Robust End-to-end Speech Recognitioncited by 5
  7. 2023Transferring speech-generic and depression-specific knowledge for Alzheimer's disease detectioncited by 4
  8. 2024Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Modelscited by 4
  9. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillationcited by 4
  10. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  11. 2022Bridging Speech and Textual Pre-trained Models with Unsupervised ASRcited by 3
  12. 2024Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Trainingcited by 3
  13. 2024Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNetcited by 3
  14. 2022Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognitioncited by 2
  15. 2022ESSumm: Extractive Speech Summarization from Untranscribed Meetingcited by 2
  16. 2024Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognitioncited by 2
  17. 2024CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processingcited by 2
  18. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 2
  19. 2022Don't Be So Sure! Boosting ASR Decoding via Confidence Relaxationcited by 1
  20. 2023Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understandingcited by 1
  21. 2024UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Modelscited by 1
  22. 2024Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Maskcited by 1
  23. 2022Alternate Intermediate Conditioning with Syllable-level and Character-level Targets for Japanese ASRcited by 0
  24. 2022Application of Knowledge Distillation to Multi-task Speech Representation Learningcited by 0
  25. 2023Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representationcited by 0
  26. 2023HuBERTopic: Enhancing Semantic Representation of HuBERT through Self-supervision Utilizing Topic Modelcited by 0
  27. 2024An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutionscited by 0