A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement
Not on the map. Why?
Abstract (arXiv)
Self-supervised learned models have been found to be very effective for certain speech tasks such as automatic speech recognition, speaker identification, keyword spotting and others. While the features are undeniably useful in speech recognition and associated tasks, their utility in speech enhancement systems is yet to be firmly established, and perhaps not properly understood. In this paper, we investigate the uses of SSL representations for single-channel speech enhancement in challenging conditions and find that they add very little value for the enhancement task. Our constraints are designed around on-device real-time speech enhancement -- model is causal, the compute footprint is small. Additionally, we focus on low SNR conditions where such models struggle to provide good enhancement. In order to systematically examine how SSL representations impact performance of such enhancement models, we propose a variety of techniques to utilize these embeddings which include different forms of knowledge-distillation and pre-training.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 3 Mar 2024.
Builds on
29 of 38 references.
References · 29
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2016Language Modeling with Gated Convolutional Networkscited by 702
- 2018SDR - half-baked or well done?cited by 423
- 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
- 2017Deep Complex Networkscited by 217
- 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 197
- 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
- 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 141
17 more
- 2022Conditional Diffusion Probabilistic Model for Speech Enhancementcited by 132
- 2019Music Source Separation in the Waveform Domaincited by 99
- 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
- 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
- 2019Phase-aware Speech Enhancement with Deep Complex U-Netcited by 73
- 2020FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancementcited by 68
- 2021Interspeech 2021 Deep Noise Suppression Challengecited by 47
- 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
- 2022Boosting Self-Supervised Embeddings for Speech Enhancementcited by 24
- 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
- 2022RemixIT: Continual self-training of speech enhancement models via bootstrapped remixingcited by 20
- 2021Noisy-target Training: A Training Strategy for DNN-based Speech Enhancement without Clean Speechcited by 15
- 2021DPCRN: Dual-Path Convolution Recurrent Network for Single Channel Speech Enhancementcited by 14
- 2022Self-Supervised Learning for Speech Enhancement through Synthesiscited by 10
- 2020Incorporating Broad Phonetic Information for Speech Enhancementcited by 5
- 2022A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancementcited by 5
- 2021Incorporating Real-world Noisy Speech in Neural-network-based Speech Enhancement Systemscited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.