# Unsupervised Speech Enhancement via Drifting

arXiv: https://arxiv.org/abs/2609.34662 · HTML: https://mlsanity.com/paper/2609.34662/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 19 of 19)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Comparative layer-wise analysis of self-supervised speech models](https://mlsanity.com/paper/2211.03929/) (2022)
- [Generative Modeling via Drifting](https://mlsanity.com/paper/2602.04770/) (2026)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [SCOREQ: Speech Quality Assessment with Contrastive Regression](https://mlsanity.com/paper/2410.06675/) (2024)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [MetricGAN-U: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech](https://mlsanity.com/paper/2110.05866/) (2021)
- [One-Step Generative Modeling via Wasserstein Gradient Flows](https://mlsanity.com/paper/2605.11755/) (2026)
- [Generative Drifting is Secretly Score Matching: a Spectral and Variational Perspective](https://mlsanity.com/paper/2603.09936/) (2026)
- [Towards Automatic Assessment of Self-Supervised Speech Models using Rank](https://mlsanity.com/paper/2409.10787/) (2024)
- [Speech Enhancement Based on Drifting Models](https://mlsanity.com/paper/2604.24199/) (2026)
- [Diffusion-based Frameworks for Unsupervised Speech Enhancement](https://mlsanity.com/paper/2601.09931/) (2026)
- [Joint magnitude estimation and phase recovery using Cycle-in-Cycle GAN for non-parallel speech enhancement](https://mlsanity.com/paper/2109.12591/) (2021)
- [DriftSE: Speech Enhancement with Generative Drifting](https://mlsanity.com/paper/2609.12252/) (2026)

## Abstract (arXiv)

This paper addresses unsupervised speech enhancement in the unpaired setting using drifting methods, where training relies on separate collections of degraded and clean audio without corresponding pairs. While recent drifting approaches enable unpaired training, they do so at a heavy cost: because the objective optimizes only a marginal prior over clean speech, the enhancer gradually loses the input's linguistic content and speaker identity. To fix this, we introduce input-conditioned drifting. We preserve the pull of the clean corpus while re-tethering the output to the degraded input via two mechanisms: an anchor encoder supplies the missing likelihood by pulling toward the input's features, and a key encoder conditions the prior by re-weighting retrieved frames. Neither requires labels or paired data. Using a training-free encoder selection criterion, Word Error Rate on VoiceBank-DEMAND falls to 10.1% (unprocessed: 11.7%), speaker similarity recovers from 0.490 to 0.879, and the recipe transfers in part to dereverberation on WSJ0-REVERB: content improves, rendering quality does not.