mlsanityData to 7 Oct 2026

MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting

arXiv 2601.20300 ↗First posted 28 Jan 2026cs.CL eess.AScited by 0

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

Self-supervised learning (SSL) has greatly advanced speech representation learning, but multilingual SSL models remain constrained to languages encountered during pretraining. Retraining from scratch to incorporate new languages is computationally expensive, while sequential training without migitation strategies often leads to catastrophic forgetting. To address this, we propose MiLorE-SSL, a lightweight framework that combines LoRA modules with a soft mixture-of-experts (MoE) mechanism for efficient continual multilingual training. LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference. To further mitigate forgetting, we introduce limited replay data from existing languages, avoiding reliance on large historical corpora. Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 25 of its 28 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

25 of 28 references.

References · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  12. 2018Experience Replay for Continual Learningcited by 1,074
13 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  3. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  4. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  5. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  6. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  7. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  8. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 64
  9. 2025Overcoming catastrophic forgetting in neural networkscited by 52
  10. 2015THCHS-30 : A Free Chinese Speech Corpuscited by 43
  11. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  12. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  13. 2024Seamless Language Expansion: Enhancing Multilingual Mastery in Self-Supervised Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing
  2. 2024MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting
  3. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
  4. 2026Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation
  5. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
  6. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
  7. 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
  8. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts