mlsanityData to 14 Oct 2026

Bridging Self-Supervised Learning and Speech Enhancement: A Wav2Vec2-Conditioned Framework

arXiv 2606.22591 ↗First posted 21 Jun 2026eess.AS cs.SDcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 23 of its 33 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

23 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation22 Dec
  2. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  3. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar
  4. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb

Other references · 19

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2018SDR - half-baked or well done?cited by 423
  12. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
7 more
  1. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  2. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  3. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  4. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  5. 2022Boosting Self-Supervised Embeddings for Speech Enhancementcited by 24
  6. 2024FINALLY: fast and universal speech enhancement with studio-like qualitycited by 7
  7. 2024GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Posterior Transition Modeling for Unsupervised Diffusion-Based Speech Enhancement
  2. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learning
  3. 2026Absorbing Discrete Diffusion for Speech Enhancement
  4. 2025Model as Loss: A Self-Consistent Training Paradigm
  5. 2024Pre-training Feature Guided Diffusion Model for Speech Enhancement
  6. 2025Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement
  7. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement
  8. 2023Diffusion-based speech enhancement with a weighted generative-supervised learning loss