Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clients

arXiv 2405.17376 ↗First posted 27 May 2024cs.CLcited by 1

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Automatic speech recognition models require large amounts of speech recordings for training. However, the collection of such data often is cumbersome and leads to privacy concerns. Federated learning has been widely used as an effective decentralized technique that collaboratively learns a shared prediction model while keeping the data local on different clients. Unfortunately, client devices often feature limited computation and communication resources leading to practical difficulties for large models. In addition, the heterogeneity that characterizes edge devices makes it sub-optimal to generate a single model that fits all of them. Differently from the recent literature, where multiple models with different architectures are used, in this work, we propose using dynamical architectures which, employing early-exit solutions, can adapt their processing (i.e. traversed layers) depending on the input and on the operation conditions. This solution falls in the realm of partial training methods and brings two benefits: a single model is used on a variety of devices; federating the models after local training is straightforward. Experiments on public datasets show that our proposed approach is effective and can be combined with basic federated learning strategies.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 59 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

43 of 59 references.

Closest ancestors · 1

Same direction, newest first

2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges8 Mar

Other references · 42

2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
2020Adaptive Federated Optimizationcited by 1,036
2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
2021Model-Contrastive Federated Learningcited by 762
2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
2021Towards Personalized Federated Learningcited by 462
2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
2020Flower: A Friendly Federated Learning Research Frameworkcited by 449

30 more

2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
2021When the Curious Abandon Honesty: Federated Learning Is Not Privatecited by 126
2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
2020Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computingcited by 52
2023A Review of Deep Learning Techniques for Speech Processingcited by 52
2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
2022Federated Learning for Healthcare Domain - Pipeline, Applications and Challengescited by 34
2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
2022On-device Training: A First Overview on Existing Systemscited by 18
2023Efficient and Secure Federated Learning for Financial Applicationscited by 17
2021PhiNets: a scalable backbone for low-power AI at the edgecited by 16
2021Conformer-based Hybrid ASR System for Switchboard Datasetcited by 15
2021Enabling On-Device Training of Speech Recognition Models with Federated Dropoutcited by 12
2021Privacy attacks for automatic speech recognition acoustic models in a federated learning frameworkcited by 10
2021Towards Understanding the Impact of Model Size on Differential Private Classificationcited by 10
2022Federated Self-supervised Speech Representations: Are We There Yet?cited by 10
2022FLAME: Federated Learning Across Multi-device Environmentscited by 8
2023Accelerating Fair Federated Learning: Adaptive Federated Adamcited by 8
2023Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASRcited by 8
2023Federated Learning for ASR based on Wav2vec 2.0cited by 7
2022Sotto Voce: Federated Speech Recognition with Differential Privacy Guaranteescited by 3
2023Improving Scheduled Sampling for Neural Transducer-based ASRcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

2026SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studiescited by 0

Similar, not cited

Close in wording, no citation either way