mlsanityData to 7 Oct 2026

Unsupervised pretraining transfers well across languages

arXiv 2002.02848 ↗First posted 7 Feb 2020eess.AS cs.CL cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting. This assumes the existence of a parallel corpus of speech and orthographic transcriptions. Recently, contrastive predictive coding (CPC) algorithms have been proposed to pretrain ASR systems with unlabelled data. In this work, we investigate whether unsupervised pretraining transfers well across languages. We show that a slight modification of the CPC pretraining extracts features that transfer well to other languages, being on par or even outperforming supervised pretraining. This shows the potential of unsupervised methods for languages with few linguistic resources.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 23; 2023: 15; 2024: 11; 2025: 8; 2026: 7.

100 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  7. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  8. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  9. 2021Unsupervised Speech Recognitioncited by 147
  10. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  12. 2021SiT: Self-supervised vIsion Transformercited by 99
38 more
  1. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  2. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 76
  3. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  6. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  7. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  8. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  9. 2023Analysing Discrete Self Supervised Speech Representation for Spoken Language Modelingcited by 47
  10. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  11. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 44
  12. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 43
  13. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  14. 2021Multi-Format Contrastive Learning of Audio Representationscited by 39
  15. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  16. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  17. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  18. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  19. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  20. 2022Do self-supervised speech models develop human-like perception biases?cited by 28
  21. 2021Self-supervised Representation Learning with Relative Predictive Codingcited by 27
  22. 2024Multilingual Turn-taking Prediction Using Voice Activity Projectioncited by 26
  23. 2021Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learningcited by 25
  24. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  25. 2023ProsAudit, a prosodic benchmark for self-supervised speech modelscited by 24
  26. 2021CLSRIL-23: Cross Lingual Speech Representations for Indic Languagescited by 22
  27. 2022XTREME-S: Evaluating Cross-lingual Speech Representationscited by 22
  28. 2024Real-time and Continuous Turn-taking Prediction Using Voice Activity Projectioncited by 21
  29. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  30. 2021Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Userscited by 16
  31. 2021Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learningcited by 16
  32. 2021Membership Inference Attacks Against Self-supervised Speech Modelscited by 16
  33. 2024On the social bias of speech self-supervised modelscited by 16
  34. 2021Aligned Contrastive Predictive Codingcited by 15
  35. 2021Self-Supervised Learning of Audio Representations from Permutations with Differentiable Rankingcited by 14
  36. 2022Speaker Normalization for Self-supervised Speech Emotion Recognitioncited by 14
  37. 2022Variable-rate hierarchical CPC leads to acoustic unit discovery in speechcited by 13
  38. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13

50 most cited of 100 All 100 (JSON)