mlsanityData to 7 Oct 2026

Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks

arXiv 2401.10070 ↗First posted 18 Jan 2024cs.CL cs.SD eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems, including automatic speech recognition (ASR) and speech translation (ST). However, the commonly used FL approach (i.e., textscFedAvg) in S2T tasks typically suffers from extensive communication overhead due to multi-round interactions based on the whole model and performance degradation caused by data heterogeneity among clients.To address these issues, we propose a personalized federated S2T framework that introduces textscFedLoRA, a lightweight LoRA module for client-side tuning and interaction with the server to minimize communication overhead, and textscFedMem, a global model equipped with a k-nearest-neighbor (kNN) classifier that captures client-specific distributional shifts to achieve personalization and overcome data heterogeneity. Extensive experiments based on Conformer and Whisper backbone models on CoVoST and GigaSpeech benchmarks show that our approach significantly reduces the communication overhead on all S2T tasks and effectively personalizes the global model to overcome data heterogeneity.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

26 of 30 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  8. 2020Nearest Neighbor Machine Translationcited by 209
  9. 2021Adaptive Nearest Neighbor Machine Translationcited by 58
  10. 2021Personalized Federated Learning through Local Memorizationcited by 57
  11. 2020Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognitioncited by 56
  12. 2020Training Speech Recognition Models with Federated Learning: A Quality/Cost Frameworkcited by 46
14 more
  1. 2021Fast Nearest Neighbor Machine Translationcited by 37
  2. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
  3. 2022Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibrationcited by 24
  4. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  5. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  6. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  7. 2023Simple and Scalable Nearest Neighbor Machine Translationcited by 12
  8. 2023Federated Learning for ASR based on Wav2vec 2.0cited by 7
  9. 2022Federated Domain Adaptation for ASR with Full Self-Supervisioncited by 6
  10. 2022Decoupled Federated Learning for ASR with Non-IID Datacited by 6
  11. 2022FedNST: Federated Noisy Student Training for Automatic Speech Recognitioncited by 5
  12. 2023Federated Nearest Neighbor Machine Translationcited by 5
  13. 2023Federated Self-Learning with Weak Supervision for Speech Recognitioncited by 5
  14. 2023Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 2.

8 papers.

  1. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  2. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  3. 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
  4. 2025An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Datacited by 4
  5. 2025Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognitioncited by 1
  6. 2025ZK-APEX: Zero-Knowledge Approximate Personalized Unlearning with Executable Proofscited by 1
  7. 2026Towards Personalized Federated Learning for Dysarthric Speech Recognitioncited by 0
  8. 2026Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarkingcited by 0