# Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0

arXiv: https://arxiv.org/abs/2110.03560 · HTML: https://mlsanity.com/paper/2110.03560/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training](https://mlsanity.com/paper/2011.13439/) (2020)
- [Transfer Learning for Speech and Language Processing](https://mlsanity.com/paper/1511.06066/) (2015)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans](https://mlsanity.com/paper/2012.13006/) (2020)
- [Exploiting semi-supervised training through a dropout regularization in end-to-end speech recognition](https://mlsanity.com/paper/1908.05227/) (2019)

## Built on by (most cited first, 12 of 12)

- [Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation](https://mlsanity.com/paper/2305.10951/) (2023)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages](https://mlsanity.com/paper/2305.12606/) (2023)
- [Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining](https://mlsanity.com/paper/2301.07295/) (2023)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek](https://mlsanity.com/paper/2301.00304/) (2022)
- [Unsupervised domain adaptation for speech recognition with unsupervised error correction](https://mlsanity.com/paper/2209.12043/) (2022)
- [Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition](https://mlsanity.com/paper/2312.12783/) (2023)
- [On Unsupervised Uncertainty-Driven Speech Pseudo-Label Filtering and Model Calibration](https://mlsanity.com/paper/2211.07795/) (2022)
- [On the Cross-lingual Transferability of Pre-trained wav2vec2-based Models](https://mlsanity.com/paper/2511.21704/) (2025)
- [Federated Representation Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2308.02013/) (2023)
- [Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR](https://mlsanity.com/paper/2604.06487/) (2026)

## Abstract (arXiv)

We propose a simple and effective cross-lingual transfer learning method to adapt monolingual wav2vec-2.0 models for Automatic Speech Recognition (ASR) in resource-scarce languages. We show that a monolingual wav2vec-2.0 is a good few-shot ASR learner in several languages. We improve its performance further via several iterations of Dropout Uncertainty-Driven Self-Training (DUST) by using a moderate-sized unlabeled speech dataset in the target language. A key finding of this work is that the adapted monolingual wav2vec-2.0 achieves similar performance as the topline multilingual XLSR model, which is trained on fifty-three languages, on the target language ASR task.