# SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies

arXiv: https://arxiv.org/abs/2607.25716 · HTML: https://mlsanity.com/paper/2607.25716/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 30 of 39)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [WavLLM: Towards Robust and Adaptive Speech Large Language Model](https://mlsanity.com/paper/2404.00656/) (2024)
- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [Hierarchical Federated Learning with Quantization: Convergence Analysis and System Design](https://mlsanity.com/paper/2103.14272/) (2021)
- [To Talk or to Work: Flexible Communication Compression for Energy Efficient Federated Learning over Heterogeneous Mobile Edge Devices](https://mlsanity.com/paper/2012.11804/) (2020)
- [LLM-PBE: Assessing Data Privacy in Large Language Models](https://mlsanity.com/paper/2408.12787/) (2024)
- [FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization](https://mlsanity.com/paper/2111.14655/) (2021)
- [End-to-End Speech Recognition from Federated Acoustic Models](https://mlsanity.com/paper/2104.14297/) (2021)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Federated Learning in ASR: Not as Easy as You Think](https://mlsanity.com/paper/2109.15108/) (2021)
- [Federated Continual Learning: Concepts, Challenges, and Solutions](https://mlsanity.com/paper/2502.07059/) (2025)
- [How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?](https://mlsanity.com/paper/2502.14502/) (2025)
- [Recurrent Early Exits for Federated Learning with Heterogeneous Clients](https://mlsanity.com/paper/2405.14791/) (2024)

## Abstract (arXiv)

Federated learning (FL) enables privacy-preserving training of automatic speech recognition (ASR) systems across distributed data sources, yet its application to large-scale speech language models (SpeechLLMs) remains unexplored. This paper presents the first systematic study of federated training for SpeechLLM-based end-to-end ASR systems. We design a communication-efficient federated optimization strategy tailored to the unique challenges of SpeechLLM architectures, addressing high-dimensional parameter spaces, gradient communication overhead, and computational constraints in distributed settings. Through extensive empirical evaluation on monolingual ASR tasks in English and Italian, we demonstrate the effectiveness and stability of our federated approach compared to centralized training baselines across diverse acoustic conditions and speaking styles. Additionally, we conduct a comprehensive ablation study analyzing the impact of different speech encoder architectures on monolingual English ASR performance within the federated framework, providing insights into optimal model configurations for decentralized training. Our results achieve competitive word error rates while reducing communication costs, establishing practical foundations for federated SpeechLLM deployment in real-world multilingual scenarios.