# Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Models

arXiv: https://arxiv.org/abs/2406.14092 · HTML: https://mlsanity.com/paper/2406.14092/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting](https://mlsanity.com/paper/1802.02950/) (2018)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data](https://mlsanity.com/paper/2209.15329/) (2022)
- [SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts](https://mlsanity.com/paper/2306.02207/) (2023)
- [Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detection](https://mlsanity.com/paper/2308.03300/) (2023)
- [Overcoming catastrophic forgetting problem by weight consolidation and long-term memory](https://mlsanity.com/paper/1805.07441/) (2018)
- [A Multi-Task Learning Framework for Overcoming the Catastrophic Forgetting in Automatic Speech Recognition](https://mlsanity.com/paper/1904.08039/) (2019)

## Built on by (most cited first, 4 of 4)

- [AfriHuBERT: A self-supervised speech representation model for African languages](https://mlsanity.com/paper/2409.20201/) (2024)
- [Enhancing Code-switched Text-to-Speech Synthesis Capability in Large Language Models with only Monolingual Corpora](https://mlsanity.com/paper/2409.10969/) (2024)
- [MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting](https://mlsanity.com/paper/2601.20300/) (2026)
- [Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgetting](https://mlsanity.com/paper/2602.12746/) (2026)

## Abstract (arXiv)

Self-supervised (SSL) models have shown great performance in various downstream tasks. However, they are typically developed for limited languages, and may encounter new languages in real-world. Developing a SSL model for each new language is costly. Thus, it is vital to figure out how to efficiently adapt existed SSL models to a new language without impairing its original abilities. We propose adaptation methods which integrate LoRA to existed SSL models to extend new language. We also develop preservation strategies which include data combination and re-clustering to retain abilities on existed languages. Applied to mHuBERT, we investigate their effectiveness on speech re-synthesis task. Experiments show that our adaptation methods enable mHuBERT to be applied to a new language (Mandarin) with MOS value increased about 1.6 and the relative value of WER reduced up to 61.72%. Also, our preservation strategies ensure that the performance on both existed and new languages remains intact.