# Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data

arXiv: https://arxiv.org/abs/2105.04727 · HTML: https://mlsanity.com/paper/2105.04727/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [Voice Separation with an Unknown Number of Multiple Speakers](https://mlsanity.com/paper/2003.01531/) (2020)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)
- [D3Net: Densely connected multidilated DenseNet for music source separation](https://mlsanity.com/paper/2010.01733/) (2020)
- [PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss](https://mlsanity.com/paper/2008.04470/) (2020)
- [Compute and memory efficient universal sound source separation](https://mlsanity.com/paper/2103.02644/) (2021)
- [Unsupervised training of a deep clustering model for multichannel blind source separation](https://mlsanity.com/paper/1904.01340/) (2019)
- [Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervision](https://mlsanity.com/paper/1911.02182/) (2019)
- [Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information](https://mlsanity.com/paper/1811.01531/) (2018)
- [Speech enhancement with weakly labelled data from AudioSet](https://mlsanity.com/paper/2102.09971/) (2021)
- [Ultra-Lightweight Speech Separation via Group Communication](https://mlsanity.com/paper/2011.08397/) (2020)
- [Bootstrapping single-channel source separation via unsupervised spatial clustering on stereo mixtures](https://mlsanity.com/paper/1811.02130/) (2018)
- [Personalized Speech Enhancement through Self-Supervised Data Augmentation and Purification](https://mlsanity.com/paper/2104.02018/) (2021)
- [DESED-FL and URBAN-FL: Federated Learning Datasets for Sound Event Detection](https://mlsanity.com/paper/2102.08833/) (2021)

## Built on by (most cited first, 8 of 8)

- [Non-IID data and Continual Learning processes in Federated Learning: A long road ahead](https://mlsanity.com/paper/2111.13394/) (2021)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Anomaly Detection through Unsupervised Federated Learning](https://mlsanity.com/paper/2209.04184/) (2022)
- [Continual self-training with bootstrapped remixing for speech enhancement](https://mlsanity.com/paper/2110.10103/) (2021)
- [TinyMLOps: Operational Challenges for Widespread Edge AI Adoption](https://mlsanity.com/paper/2203.10923/) (2022)
- [Contrastive Federated Learning with Tabular Data Silos](https://mlsanity.com/paper/2409.06123/) (2024)
- [Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions](https://mlsanity.com/paper/2407.04302/) (2024)
- [FedLED: Label-Free Equipment Fault Diagnosis with Vertical Federated Transfer Learning](https://mlsanity.com/paper/2312.17451/) (2023)

## Abstract (arXiv)

We propose FEDENHANCE, an unsupervised federated learning (FL) approach for speech enhancement and separation with non-IID distributed data across multiple clients. We simulate a real-world scenario where each client only has access to a few noisy recordings from a limited and disjoint number of speakers (hence non-IID). Each client trains their model in isolation using mixture invariant training while periodically providing updates to a central server. Our experiments show that our approach achieves competitive enhancement performance compared to IID training on a single device and that we can further facilitate the convergence speed and the overall performance using transfer learning on the server-side. Moreover, we show that we can effectively combine updates from clients trained locally with supervised and unsupervised losses. We also release a new dataset LibriFSD50K and its creation recipe in order to facilitate FL research for source separation problems.