# Bridging Self-Supervised Learning and Speech Enhancement: A Wav2Vec2-Conditioned Framework

arXiv: https://arxiv.org/abs/2606.22591 · HTML: https://mlsanity.com/paper/2606.22591/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 21 Jun 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [Investigating self-supervised learning for speech enhancement and separation](https://mlsanity.com/paper/2203.07960/) (2022)
- [Boosting Self-Supervised Embeddings for Speech Enhancement](https://mlsanity.com/paper/2204.03339/) (2022)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioning](https://mlsanity.com/paper/2410.13599/) (2024)

## Abstract (arXiv)

Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.