# Bayesian Low-Rank Factorization for Robust Model Adaptation

arXiv: https://arxiv.org/abs/2510.18723 · HTML: https://mlsanity.com/paper/2510.18723/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 24 of 24)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Bayesian Low-rank Adaptation for Large Language Models](https://mlsanity.com/paper/2308.13111/) (2023)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning](https://mlsanity.com/paper/2311.11077/) (2023)
- [BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models](https://mlsanity.com/paper/2406.11675/) (2024)
- [Bayesian Recurrent Neural Networks](https://mlsanity.com/paper/1704.02798/) (2017)
- [End-to-End Speech Translation for Code Switched Speech](https://mlsanity.com/paper/2204.05076/) (2022)
- [Data Augmentation for End-to-end Code-switching Speech Recognition](https://mlsanity.com/paper/2011.02160/) (2020)
- [Efficient Weight factorization for Multilingual Speech Recognition](https://mlsanity.com/paper/2105.03010/) (2021)
- [PIER: A Novel Metric for Evaluating What Matters in Code-Switching](https://mlsanity.com/paper/2501.09512/) (2025)
- [Textual Data Augmentation for Arabic-English Code-Switching Speech Recognition](https://mlsanity.com/paper/2201.02550/) (2022)
- [End-to-End Evaluation for Low-Latency Simultaneous Speech Translation](https://mlsanity.com/paper/2308.03415/) (2023)
- [Bayesian Learning for Deep Neural Network Adaptation](https://mlsanity.com/paper/2012.07460/) (2020)
- [Adaptive multilingual speech recognition with pretrained models](https://mlsanity.com/paper/2205.12304/) (2022)
- [Investigating Lexical Replacements for Arabic-English Code-Switched Data Augmentation](https://mlsanity.com/paper/2205.12649/) (2022)
- [Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition](https://mlsanity.com/paper/2210.08992/) (2022)
- [KIT's Multilingual Speech Translation System for IWSLT 2023](https://mlsanity.com/paper/2306.05320/) (2023)
- [Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition](https://mlsanity.com/paper/2309.11327/) (2023)
- [Weight Factorization and Centralization for Continual Learning in Speech Recognition](https://mlsanity.com/paper/2506.16574/) (2025)

## Built on by (most cited first, 1 of 1)

- [Adding Robust Code-Switching Capabilities to High Performance Multilingual ASR](https://mlsanity.com/paper/2606.21990/) (2026)

## Abstract (arXiv)

Large speech foundation models achieve strong performance across many domains, but they often require adaptation to handle local needs such as code-switching, where speakers mix languages within the same utterance. Direct fine-tuning of these models risks overfitting to the target domain and overwriting the broad capabilities of the base model. To address this challenge, we explore Bayesian factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices and thereby retain general performance while adapting to specific domains. We apply our approach to the Whisper model and evaluate on different multilingual code-switching scenarios. Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model. Compared to LoRA, our method achieves a backward gain of 54% with only a 4% drop on the new domain. These findings highlight the effectiveness of Bayesian adaptation for fine-tuning speech foundation models without sacrificing generalization.