mlsanityData to 7 Oct 2026

Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction

arXiv 2001.10603 ↗First posted 28 Jan 2020eess.AS cs.SDcited by 38

Not on the map. Why?

Abstract (arXiv)

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 20; 2022: 12; 2023: 3; 2024: 3.

38 papers.

Sort:

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  4. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2023A Cookbook of Self-Supervised Learningcited by 248
  7. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  8. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  9. 2022Assessing the State of Self-Supervised Human Activity Recognition using Wearablescited by 63
  10. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  11. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  12. 2021Contrastively Disentangled Sequential Variational Autoencodercited by 34
26 more
  1. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  2. 2022RemixIT: Continual self-training of speech enhancement models via bootstrapped remixingcited by 20
  3. 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
  4. 2022SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Trainingcited by 19
  5. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  6. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  7. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  8. 2022On-demand compute reduction with stochastic wav2vec 2.0cited by 10
  9. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
  10. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  11. 2021TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretrainingcited by 8
  12. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  13. 2021Improving speech recognition models with small samples for air traffic control systemscited by 6
  14. 2021Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASRcited by 5
  15. 2021Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic modelcited by 5
  16. 2021Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL Systemcited by 4
  17. 2021On Scaling Contrastive Representations for Low-Resource Speech Recognitioncited by 3
  18. 2021Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcriptioncited by 3
  19. 2022Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusioncited by 3
  20. 2021Unsupervised Word Segmentation from Discrete Speech Units in Low-Resource Settingscited by 2
  21. 2022Compressed Predictive Information Codingcited by 2
  22. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  23. 2022Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approachcited by 1
  24. 2024Towards the Next Frontier in Speech Representation Learning Using Disentanglementcited by 1
  25. 2024PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapsecited by 1
  26. 2024CochCeps-Augment: A Novel Self-Supervised Contrastive Learning Using Cochlear Cepstrum-based Masking for Speech Emotion Recognitioncited by 0