mlsanityData to 7 Oct 2026

Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Models

arXiv 2406.14092 ↗First posted 20 Jun 2024cs.CL eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Self-supervised (SSL) models have shown great performance in various downstream tasks. However, they are typically developed for limited languages, and may encounter new languages in real-world. Developing a SSL model for each new language is costly. Thus, it is vital to figure out how to efficiently adapt existed SSL models to a new language without impairing its original abilities. We propose adaptation methods which integrate LoRA to existed SSL models to extend new language. We also develop preservation strategies which include data combination and re-clustering to retain abilities on existed languages. Applied to mHuBERT, we investigate their effectiveness on speech re-synthesis task. Experiments show that our adaptation methods enable mHuBERT to be applied to a new language (Mandarin) with MOS value increased about 1.6 and the relative value of WER reduced up to 61.72%. Also, our preservation strategies ensure that the performance on both existed and new languages remains intact.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 26 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

22 of 26 references.

References · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  9. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  11. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  12. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
10 more
  1. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  2. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  3. 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 125
  4. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  5. 2020AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselinescited by 111
  6. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  7. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  8. 2023Do You Remember? Overcoming Catastrophic Forgetting for Fake Audio Detectioncited by 11
  9. 2018Overcoming catastrophic forgetting problem by weight consolidation and long-term memorycited by 6
  10. 2019A Multi-Task Learning Framework for Overcoming the Catastrophic Forgetting in Automatic Speech Recognitioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2024AfriHuBERT: A self-supervised speech representation model for African languagescited by 17
  2. 2024Enhancing Code-switched Text-to-Speech Synthesis Capability in Large Language Models with only Monolingual Corporacited by 0
  3. 2026MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgettingcited by 0
  4. 2026Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgettingcited by 0