# Learning Robust and Multilingual Speech Representations

arXiv: https://arxiv.org/abs/2001.11128 · HTML: https://mlsanity.com/paper/2001.11128/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2020.

## Built on by (most cited first, 30 of 39)

- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://mlsanity.com/paper/2101.07597/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech](https://mlsanity.com/paper/2104.11462/) (2021)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality](https://mlsanity.com/paper/2207.07036/) (2022)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [Whodunit? Learning to Contrast for Authorship Attribution](https://mlsanity.com/paper/2209.11887/) (2022)
- [Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning](https://mlsanity.com/paper/2106.00273/) (2021)
- [Differentially Private Speaker Anonymization](https://mlsanity.com/paper/2202.11823/) (2022)
- [A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systems](https://mlsanity.com/paper/2204.01397/) (2022)
- [Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction](https://mlsanity.com/paper/2110.15430/) (2021)
- [An Evaluation of Self-Supervised Pre-Training for Skin-Lesion Analysis](https://mlsanity.com/paper/2106.09229/) (2021)
- [Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond](https://mlsanity.com/paper/2310.05513/) (2023)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [On Architectures and Training for Raw Waveform Feature Extraction in ASR](https://mlsanity.com/paper/2104.04298/) (2021)
- [Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech](https://mlsanity.com/paper/2303.11131/) (2023)
- [Self-Supervised Learning from Automatically Separated Sound Scenes](https://mlsanity.com/paper/2105.02132/) (2021)
- [ON-TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks](https://mlsanity.com/paper/2205.01987/) (2022)
- [Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budget](https://mlsanity.com/paper/2409.16295/) (2024)
- [Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Models](https://mlsanity.com/paper/2305.05201/) (2023)
- [Fast Development of ASR in African Languages using Self Supervised Speech Representation Learning](https://mlsanity.com/paper/2103.08993/) (2021)
- [Injecting Text and Cross-lingual Supervision in Few-shot Learning from Self-Supervised Models](https://mlsanity.com/paper/2110.04863/) (2021)

## Abstract (arXiv)

Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages including tonal languages and low-resource languages.