mlsanityData to 7 Oct 2026

Exploring Federated Self-Supervised Learning for General Purpose Audio Understanding

arXiv 2402.02889 ↗First posted 5 Feb 2024cs.SD cs.CV cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

The integration of Federated Learning (FL) and Self-supervised Learning (SSL) offers a unique and synergetic combination to exploit the audio data for general-purpose audio understanding, without compromising user data privacy. However, rare efforts have been made to investigate the SSL models in the FL regime for general-purpose audio understanding, especially when the training data is generated by large-scale heterogeneous audio sources. In this paper, we evaluate the performance of feature-matching and predictive audio-SSL techniques when integrated into large-scale FL settings simulated with non-independently identically distributed (non-iid) data. We propose a novel Federated SSL (F-SSL) framework, dubbed FASSL, that enables learning intermediate feature representations from large-scale decentralized heterogeneous clients, holding unlabelled audio data. Our study has found that audio F-SSL approaches perform on par with the centralized audio-SSL approaches on the audio-retrieval task. Extensive experiments demonstrate the effectiveness and significance of FASSL as it assists in obtaining the optimal global model for state-of-the-art FL aggregation methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

20 of 27 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  7. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  8. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  9. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  10. 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 68
  11. 2021Towards Learning Universal Audio Representationscited by 51
  12. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
8 more
  1. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  2. 2020Federated Self-Supervised Learning of Multi-Sensor Representations for Embedded Intelligencecited by 39
  3. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
  4. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
  5. 2022FedAudio: A Federated Learning Benchmark for Audio Taskscited by 15
  6. 2022Federated Self-supervised Learning for Video Understandingcited by 13
  7. 2022Federated Self-Supervised Learning for Acoustic Event Classificationcited by 5
  8. 2021Inferring Facing Direction from Voice Signalscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understandingcited by 1
  2. 2026Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encodercited by 0