mlsanityData to 7 Oct 2026

Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognition

arXiv 2203.16973 ↗First posted 31 Mar 2022cs.CL cs.SD eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) to learn high-level speech representations has been a popular approach to building Automatic Speech Recognition (ASR) systems in low-resource settings. However, the common assumption made in literature is that a considerable amount of unlabeled data is available for the same domain or language that can be leveraged for SSL pre-training, which we acknowledge is not feasible in a real-world setting. In this paper, as part of the Interspeech Gram Vaani ASR challenge, we try to study the effect of domain, language, dataset size, and other aspects of our upstream pre-training SSL data on the final performance low-resource downstream ASR task. We also build on the continued pre-training paradigm to study the effect of prior knowledge possessed by models trained using SSL. Extensive experiments and studies reveal that the performance of ASR systems is susceptible to the data used for SSL pre-training. Their performance improves with an increase in similarity and volume of pre-training data. We believe our work will be helpful to the speech community in building better ASR systems in low-resource settings and steer research towards improving generalization in SSL-based pre-training for speech systems.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

33 of 38 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
21 more
  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  2. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  3. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  5. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  6. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  7. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  8. 2017Searching for Activation Functionscited by 158
  9. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  10. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  11. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  12. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  13. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  14. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  15. 2021Towards Building ASR Systems for the Next Billion Userscited by 45
  16. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  17. 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
  18. 2021Multiview Pseudo-Labeling for Semi-supervised Learning from Videocited by 32
  19. 2021CLSRIL-23: Cross Lingual Speech Representations for Indic Languagescited by 22
  20. 2021Aligned Contrastive Predictive Codingcited by 15
  21. 2021The History of Speech Recognition to the Year 2030cited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1.

2 papers.

  1. 2024How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenariocited by 5
  2. 2022Self-Relation Attention and Temporal Awareness for Emotion Recognition via Vocal Burstcited by 1