mlsanityData to 14 Oct 2026

SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies

arXiv 2607.25716 ↗First posted 28 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables privacy-preserving training of automatic speech recognition (ASR) systems across distributed data sources, yet its application to large-scale speech language models (SpeechLLMs) remains unexplored. This paper presents the first systematic study of federated training for SpeechLLM-based end-to-end ASR systems. We design a communication-efficient federated optimization strategy tailored to the unique challenges of SpeechLLM architectures, addressing high-dimensional parameter spaces, gradient communication overhead, and computational constraints in distributed settings. Through extensive empirical evaluation on monolingual ASR tasks in English and Italian, we demonstrate the effectiveness and stability of our federated approach compared to centralized training baselines across diverse acoustic conditions and speaking styles. Additionally, we conduct a comprehensive ablation study analyzing the impact of different speech encoder architectures on monolingual English ASR performance within the federated framework, providing insights into optimal model configurations for decentralized training. Our results achieve competitive word error rates while reducing communication costs, establishing practical foundations for federated SpeechLLM deployment in real-world multilingual scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

39 of 54 references.

References · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2023Mistral 7Bcited by 3,164
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
27 more
  1. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  2. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  3. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  4. 2018Sparsified SGD with Memorycited by 376
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  6. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  7. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  8. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  9. 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
  10. 2020To Talk or to Work: Flexible Communication Compression for Energy Efficient Federated Learning over Heterogeneous Mobile Edge Devicescited by 46
  11. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  12. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  13. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
  14. 2021Conformer-based Hybrid ASR System for Switchboard Datasetcited by 15
  15. 2021Federated Learning in ASR: Not as Easy as You Thinkcited by 14
  16. 2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
  17. 2025How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?cited by 13
  18. 2024Recurrent Early Exits for Federated Learning with Heterogeneous Clientscited by 12
  19. 2022Federated Self-supervised Speech Representations: Are We There Yet?cited by 10
  20. 2023Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASRcited by 8
  21. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 8
  22. 2023Federated Learning for ASR based on Wav2vec 2.0cited by 7
  23. 2022Federated Domain Adaptation for ASR with Full Self-Supervisioncited by 6
  24. 2024Efficient and Personalized Mobile Health Event Prediction via Small Language Modelscited by 6
  25. 2023Improving Scheduled Sampling for Neural Transducer-based ASRcited by 3
  26. 2024Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clientscited by 1
  27. 2025SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settingscited by 1

Built on by

Not cited yet.