mlsanityData to 7 Oct 2026

Federated Self-supervised Speech Representations: Are We There Yet?

arXiv 2204.02804 ↗First posted 6 Apr 2022cs.SD cs.LG eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

The ubiquity of microphone-enabled devices has lead to large amounts of unlabelled audio data being produced at the edge. The integration of self-supervised learning (SSL) and federated learning (FL) into one coherent system can potentially offer data privacy guarantees while also advancing the quality and robustness of speech representations. In this paper, we provide a first-of-its-kind systematic study of the feasibility and complexities for training speech SSL models under FL scenarios from the perspective of algorithms, hardware, and systems limits. Despite the high potential of their combination, we find existing system constraints and algorithmic behaviour make SSL and FL systems nearly impossible to build today. Yet critically, our results indicate specific performance bottlenecks and research opportunities that would allow this situation to be reversed. While our analysis suggests that, given existing trends in hardware, hybrid SSL and FL speech systems will not be viable until 2027. We believe this study can act as a roadmap to accelerate work towards reaching this milestone much earlier.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

23 of 31 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  11. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  12. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
11 more
  1. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  2. 2021A Survey on Neural Speech Synthesiscited by 182
  3. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  4. 2020The Hardware Lotterycited by 160
  5. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  6. 2017Hello Edge: Keyword Spotting on Microcontrollerscited by 108
  7. 2018Federated Learning for Keyword Spottingcited by 95
  8. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  9. 2020Can Federated Learning Save The Planet?cited by 52
  10. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
  11. 2020Improving on-device speaker verification using federated learning with privacycited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 1; 2026: 1.

10 papers.

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  2. 2023Federated Learning for ASR based on Wav2vec 2.0cited by 7
  3. 2022Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audiocited by 6
  4. 2023HyperConformer: Multi-head HyperMixer for Efficient Speech Recognitioncited by 5
  5. 2024FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clientscited by 2
  6. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  7. 2024Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clientscited by 1
  8. 2025FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understandingcited by 1
  9. 2023Federated Representation Learning for Automatic Speech Recognitioncited by 0
  10. 2026SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studiescited by 0