# How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario

arXiv: https://arxiv.org/abs/2411.18217 · HTML: https://mlsanity.com/paper/2411.18217/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 27 of 27)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [Exploiting Adapters for Cross-lingual Low-resource Speech Recognition](https://mlsanity.com/paper/2105.11905/) (2021)
- [Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.03218/) (2022)
- [AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks](https://mlsanity.com/paper/2205.00305/) (2022)
- [Massively Multilingual Adversarial Speech Recognition](https://mlsanity.com/paper/1904.02210/) (2019)
- [Exploring Efficient-tuning Methods in Self-supervised Speech Models](https://mlsanity.com/paper/2210.06175/) (2022)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2107.13530/) (2021)
- [Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction](https://mlsanity.com/paper/2310.02720/) (2023)
- [CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models](https://mlsanity.com/paper/2212.01282/) (2022)
- [Efficient Domain Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2302.01496/) (2023)
- [Cross-lingual Transfer for Speech Processing using Acoustic Language Similarity](https://mlsanity.com/paper/2111.01326/) (2021)
- [A Novel Self-training Approach for Low-resource Speech Recognition](https://mlsanity.com/paper/2308.05269/) (2023)
- [Analyzing the factors affecting usefulness of Self-Supervised Pre-trained Representations for Speech Recognition](https://mlsanity.com/paper/2203.16973/) (2022)

## Built on by (most cited first, 5 of 5)

- [Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative](https://mlsanity.com/paper/2508.09294/) (2025)
- [Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks](https://mlsanity.com/paper/2601.12205/) (2026)
- [SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models](https://mlsanity.com/paper/2607.05365/) (2026)
- [Exploring Dynamic Parameters for Vietnamese Gender-Independent ASR](https://mlsanity.com/paper/2507.22964/) (2025)
- [Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens](https://mlsanity.com/paper/2607.26350/) (2026)

## Abstract (arXiv)

The utilization of speech Self-Supervised Learning (SSL) models achieves impressive performance on Automatic Speech Recognition (ASR). However, in low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. Typical solutions like fine-tuning the SSL model suffer from high computation costs while using frozen SSL models as feature extractors comes with poor performance. To handle these issues, we extend a conventional efficient fine-tuning scheme based on the adapter. We add an extra intermediate adaptation to warm up the adapter and downstream model initialization. Remarkably, we update only 1-5% of the total model parameters to achieve the adaptation. Experimental results on the ML-SUPERB dataset show that our solution outperforms conventional efficient fine-tuning. It achieves up to a 28% relative improvement in the Character/Phoneme error rate when adapting to unseen languages.