# RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing

arXiv: https://arxiv.org/abs/2202.08862 · HTML: https://mlsanity.com/paper/2202.08862/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 67 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 30 of 45)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [Sudo rm -rf: Efficient Networks for Universal Audio Source Separation](https://mlsanity.com/paper/2007.06833/) (2020)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction](https://mlsanity.com/paper/2001.10603/) (2020)
- [Differentiable Consistency Constraints for Improved Deep Speech Enhancement](https://mlsanity.com/paper/1811.08521/) (2018)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention](https://mlsanity.com/paper/2002.05873/) (2020)
- [PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss](https://mlsanity.com/paper/2008.04470/) (2020)
- [Convolutional-Recurrent Neural Networks for Speech Enhancement](https://mlsanity.com/paper/1805.00579/) (2018)
- [Two-Step Sound Source Separation: Training on Learned Latent Targets](https://mlsanity.com/paper/1910.09804/) (2019)

## Built on by (most cited first, 20 of 20)

- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition](https://mlsanity.com/paper/2308.10415/) (2023)
- [ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior](https://mlsanity.com/paper/2505.05657/) (2025)
- [Speech separation with large-scale self-supervised learning](https://mlsanity.com/paper/2211.05172/) (2022)
- [Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separation](https://mlsanity.com/paper/2204.03232/) (2022)
- [UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures](https://mlsanity.com/paper/2305.20054/) (2023)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Reverberation as Supervision for Speech Separation](https://mlsanity.com/paper/2211.08303/) (2022)
- [On monoaural speech enhancement for automatic recognition of real noisy speech using mixture invariant training](https://mlsanity.com/paper/2205.01751/) (2022)
- [A Training and Inference Strategy Using Noisy and Enhanced Speech as Target for Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2210.15368/) (2022)
- [A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement](https://mlsanity.com/paper/2403.01369/) (2024)
- [Speech Separation with Pretrained Frontend to Minimize Domain Mismatch](https://mlsanity.com/paper/2411.03085/) (2024)
- [Speech Separation based on Contrastive Learning and Deep Modularization](https://mlsanity.com/paper/2305.10652/) (2023)
- [Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge](https://mlsanity.com/paper/2402.01413/) (2024)
- [Spectron: Target Speaker Extraction using Conditional Transformer with Adversarial Refinement](https://mlsanity.com/paper/2409.01352/) (2024)
- [Unsupervised Single-Channel Speech Separation with Diffusion under Speaker-Embedding Guidance](https://mlsanity.com/paper/2509.24395/) (2025)
- [Universal Robust Speech Adaptation for Cross-Domain Speech Recognition and Enhancement](https://mlsanity.com/paper/2602.04307/) (2026)
- [Towards Lightweight Adaptation of Speech Enhancement Models in Real-World Environments](https://mlsanity.com/paper/2603.07471/) (2026)
- [SURF: Separation via Unsupervised Remixing Flow](https://mlsanity.com/paper/2606.04921/) (2026)
- [Test-time adaptation for speech enhancement with an autoregressive speech prior](https://mlsanity.com/paper/2609.03622/) (2026)

## Abstract (arXiv)

We present RemixIT, a simple yet effective self-supervised method for training speech enhancement without the need of a single isolated in-domain speech nor a noise waveform. Our approach overcomes limitations of previous methods which make them dependent on clean in-domain target signals and thus, sensitive to any domain mismatch between train and test samples. RemixIT is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures. Then, by permuting the estimated clean and noise signals and remixing them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network. Vice-versa, the teacher periodically refines its estimates using the updated parameters of the latest student models. Experimental results on multiple speech enhancement datasets and tasks not only show the superiority of our method over prior approaches but also showcase that RemixIT can be combined with any separation model as well as be applied towards any semi-supervised and unsupervised domain adaptation task. Our analysis, paired with empirical evidence, sheds light on the inside functioning of our self-training scheme wherein the student model keeps obtaining better performance while observing severely degraded pseudo-targets.