mlsanityData to 14 Oct 2026

Self-Supervised Learning for Speech Enhancement through Synthesis

arXiv 2211.02542 ↗First posted 4 Nov 2022eess.AS cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Modern speech enhancement (SE) networks typically implement noise suppression through time-frequency masking, latent representation masking, or discriminative signal prediction. In contrast, some recent works explore SE via generative speech synthesis, where the system's output is synthesized by a neural vocoder after an inherently lossy feature-denoising step. In this paper, we propose a denoising vocoder (DeVo) approach, where a vocoder accepts noisy representations and learns to directly synthesize clean speech. We leverage rich representations from self-supervised learning (SSL) speech models to discover relevant features. We conduct a candidate search across 15 potential SSL front-ends and subsequently train our vocoder adversarially with the best SSL configuration. Additionally, we demonstrate a causal version capable of running on streaming audio with 10ms latency and minimal performance degradation. Finally, we conduct both objective evaluations and subjective listening studies to show our system improves objective metrics and outperforms an existing state-of-the-art SE model subjectively.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

28 of 38 references.

References · 28

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  9. 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
  10. 2021DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressorscited by 172
  11. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 133
  12. 2020Unsupervised pretraining transfers well across languagescited by 101
16 more
  1. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  2. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  3. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 45
  4. 2020TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aidscited by 42
  5. 2022BYOL for Audio: Exploring Pre-trained General-purpose Audio Representationscited by 36
  6. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  7. 2018Speech Denoising with Deep Feature Lossescited by 26
  8. 2022Boosting Self-Supervised Embeddings for Speech Enhancementcited by 24
  9. 2021SERAB: A multi-lingual benchmark for speech emotion recognitioncited by 22
  10. 2021Generative Speech Coding with Predictive Variance Regularizationcited by 18
  11. 2020Dense CNN with Self-Attention for Time-Domain Speech Enhancementcited by 16
  12. 2021High Fidelity Speech Regeneration with Application to Speech Enhancementcited by 10
  13. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 10
  14. 2021Weight, Block or Unit? Exploring Sparsity Tradeoffs for Speech Enhancement on Tiny Neural Acceleratorscited by 8
  15. 2019An Investigation into the Effectiveness of Enhancement in ASR Training and Test for CHiME-5 Dinner Party Transcriptioncited by 7
  16. 2022Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Loadcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 4; 2025: 1; 2026: 1.

10 papers.

  1. 2023Self-Supervised Visual Acoustic Matchingcited by 14
  2. 2024BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embeddingcited by 10
  3. 2023Two-Step Knowledge Distillation for Tiny Speech Enhancementcited by 9
  4. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancementcited by 5
  5. 2025PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancementcited by 5
  6. 2022Deep neural network techniques for monaural speech enhancement: state of the art analysiscited by 3
  7. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  8. 2024Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpaintingcited by 2
  9. 2024A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancementcited by 1
  10. 2026OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSLcited by 0