# Continual self-training with bootstrapped remixing for speech enhancement

arXiv: https://arxiv.org/abs/2110.10103 · HTML: https://mlsanity.com/paper/2110.10103/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [SlimIPL: Language-Model-Free Iterative Pseudo-Labeling](https://mlsanity.com/paper/2010.11524/) (2020)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)
- [PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss](https://mlsanity.com/paper/2008.04470/) (2020)
- [Dense CNN with Self-Attention for Time-Domain Speech Enhancement](https://mlsanity.com/paper/2009.01941/) (2020)
- [Noisy-target Training: A Training Strategy for DNN-based Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2101.08625/) (2021)
- [Compute and memory efficient universal sound source separation](https://mlsanity.com/paper/2103.02644/) (2021)
- [Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation](https://mlsanity.com/paper/2105.03544/) (2021)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [Ultra-Lightweight Speech Separation via Group Communication](https://mlsanity.com/paper/2011.08397/) (2020)
- [Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data](https://mlsanity.com/paper/2105.04727/) (2021)
- [Training Speech Enhancement Systems with Noisy Speech Datasets](https://mlsanity.com/paper/2105.12315/) (2021)
- [Semi-Supervised Singing Voice Separation with Noisy Self-Training](https://mlsanity.com/paper/2102.07961/) (2021)
- [Personalized Speech Enhancement through Self-Supervised Data Augmentation and Purification](https://mlsanity.com/paper/2104.02018/) (2021)
- [Incorporating Real-world Noisy Speech in Neural-network-based Speech Enhancement Systems](https://mlsanity.com/paper/2109.05172/) (2021)

## Built on by (most cited first, 6 of 6)

- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [MixCycle: Unsupervised Speech Separation via Cyclic Mixture Permutation Invariant Training](https://mlsanity.com/paper/2202.03875/) (2022)
- [A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancement](https://mlsanity.com/paper/2206.11000/) (2022)
- [On monoaural speech enhancement for automatic recognition of real noisy speech using mixture invariant training](https://mlsanity.com/paper/2205.01751/) (2022)
- [A Training and Inference Strategy Using Noisy and Enhanced Speech as Target for Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2210.15368/) (2022)
- [Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge](https://mlsanity.com/paper/2402.01413/) (2024)

## Abstract (arXiv)

We propose RemixIT, a simple and novel self-supervised training method for speech enhancement. The proposed method is based on a continuously self-training scheme that overcomes limitations from previous studies including assumptions for the in-domain noise distribution and having access to clean target signals. Specifically, a separation teacher model is pre-trained on an out-of-domain dataset and is used to infer estimated target signals for a batch of in-domain mixtures. Next, we bootstrap the mixing process by generating artificial mixtures using permuted estimated clean and noise signals. Finally, the student model is trained using the permuted estimated sources as targets while we periodically update teacher's weights using the latest student model. Our experiments show that RemixIT outperforms several previous state-of-the-art self-supervised methods under multiple speech enhancement tasks. Additionally, RemixIT provides a seamless alternative for semi-supervised and unsupervised domain adaptation for speech enhancement tasks, while being general enough to be applied to any separation task and paired with any separation model.