# Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation

arXiv: https://arxiv.org/abs/2502.17380 · HTML: https://mlsanity.com/paper/2502.17380/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 55 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 30 of 44)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [Representation Surgery for Multi-Task Model Merging](https://mlsanity.com/paper/2402.02705/) (2024)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Optimizing Mode Connectivity via Neuron Alignment](https://mlsanity.com/paper/2009.02439/) (2020)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)

## Built on by (most cited first, 6 of 6)

- [PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference](https://mlsanity.com/paper/2511.04805/) (2025)
- [DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2505.15090/) (2025)
- [Pseudo2Real: Task Arithmetic for Pseudo-Label Correction in Automatic Speech Recognition](https://mlsanity.com/paper/2510.08047/) (2025)
- [Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR](https://mlsanity.com/paper/2603.05354/) (2026)
- [Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition](https://mlsanity.com/paper/2603.17558/) (2026)
- [BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models](https://mlsanity.com/paper/2609.09554/) (2026)

## Abstract (arXiv)

Language diversity presents a significant challenge in speech-to-text (S2T) tasks, such as automatic speech recognition and translation. Traditional multi-lingual multi-task training approaches aim to address this by jointly optimising multiple speech recognition and translation tasks across various languages. While models like Whisper, built on these strategies, demonstrate strong performance, they still face issues of high computational cost, language interference, suboptimal training configurations, and limited extensibility. To overcome these challenges, we introduce LoRS-Merging (low-rank and sparse model merging), a novel technique designed to efficiently integrate models trained on different languages or tasks while preserving performance and reducing computational overhead. LoRS-Merging combines low-rank and sparse pruning to retain essential structures while eliminating redundant parameters, mitigating language interference, and enhancing extensibility. Experimental results across 10 languages demonstrate that LoRS-Merging significantly outperforms multi-lingual multi-task training, sequential training, and other merging methods, achieving over 20% improvement in normalised performance. Our findings suggest that model merging, particularly LoRS-Merging, is a scalable and effective complement to traditional multi-lingual training strategies for S2T applications.