# Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining

arXiv: https://arxiv.org/abs/2301.07295 · HTML: https://mlsanity.com/paper/2301.07295/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 18 Jan 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition](https://mlsanity.com/paper/2109.06870/) (2021)
- [JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis](https://mlsanity.com/paper/1711.00354/) (2017)
- [How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://mlsanity.com/paper/2106.02124/) (2021)
- [XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering](https://mlsanity.com/paper/1905.11471/) (2019)
- [Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0](https://mlsanity.com/paper/2110.03560/) (2021)
- [On-demand compute reduction with stochastic wav2vec 2.0](https://mlsanity.com/paper/2204.11934/) (2022)
- [Wav2Vec-Aug: Improved self-supervised training with limited data](https://mlsanity.com/paper/2206.13654/) (2022)
- [Speech Corpus of Ainu Folklore and End-to-end Speech Recognition for Ainu Language](https://mlsanity.com/paper/2002.06675/) (2020)

## Built on by (most cited first, 18 of 18)

- [Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation](https://mlsanity.com/paper/2305.10951/) (2023)
- [Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens](https://mlsanity.com/paper/2402.02302/) (2024)
- [Cyberbullying Detection for Low-resource Languages and Dialects: Review of the State of the Art](https://mlsanity.com/paper/2308.15745/) (2023)
- [CPT-Boosted Wav2vec2.0: Towards Noise Robust Speech Recognition for Classroom Environments](https://mlsanity.com/paper/2409.14494/) (2024)
- [In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties](https://mlsanity.com/paper/2505.14887/) (2025)
- [Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages](https://mlsanity.com/paper/2506.17459/) (2025)
- [Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Pooling](https://mlsanity.com/paper/2306.02153/) (2023)
- [CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning](https://mlsanity.com/paper/2506.17818/) (2025)
- [Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data](https://mlsanity.com/paper/2603.11378/) (2026)
- [Automatic Speech Recognition for the Ika Language](https://mlsanity.com/paper/2410.00940/) (2024)
- [SimClass: A Classroom Speech Dataset Generated via Game Engine Simulation For Automatic Speech Recognition Research](https://mlsanity.com/paper/2506.09206/) (2025)
- [Linguistically Informed Tokenization Improves ASR for Underresourced Languages](https://mlsanity.com/paper/2510.06461/) (2025)
- [Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment](https://mlsanity.com/paper/2406.05784/) (2024)
- [Non-autoregressive Personalized Bundle Generation](https://mlsanity.com/paper/2406.06925/) (2024)
- [Exploring the Impact of Data Quantity on ASR in Extremely Low-resource Languages](https://mlsanity.com/paper/2409.08872/) (2024)
- [RealClass: A Framework for Classroom Speech Simulation with Public Datasets and Game Engines](https://mlsanity.com/paper/2510.01462/) (2025)
- [Evaluating the Effect of Linguistic Relatedness on Cross-Lingual Transfer in Large Multilingual Automatic Speech Recognition](https://mlsanity.com/paper/2607.04814/) (2026)
- [From Sinhala to Dhivehi: Cross-Lingual Transfer Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2607.06289/) (2026)

## Abstract (arXiv)

In recent years, neural models learned through self-supervised pretraining on large scale multilingual text or speech data have exhibited promising results for underresourced languages, especially when a relatively large amount of data from related language(s) is available. While the technology has a potential for facilitating tasks carried out in language documentation projects, such as speech transcription, pretraining a multilingual model from scratch for every new language would be highly impractical. We investigate the possibility for adapting an existing multilingual wav2vec 2.0 model for a new language, focusing on actual fieldwork data from a critically endangered tongue: Ainu. Specifically, we (i) examine the feasibility of leveraging data from similar languages also in fine-tuning; (ii) verify whether the model's performance can be improved by further pretraining on target language data. Our results show that continued pretraining is the most effective method to adapt a wav2vec 2.0 model for a new language and leads to considerable reduction in error rates. Furthermore, we find that if a model pretrained on a related speech variety or an unrelated language with similar phonological characteristics is available, multilingual fine-tuning using additional data from that language can have positive impact on speech recognition performance when there is very little labeled data in the target language.