# Boosting Self-Supervised Embeddings for Speech Enhancement

arXiv: https://arxiv.org/abs/2204.03339 · HTML: https://mlsanity.com/paper/2204.03339/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2022.

## Built on by (most cited first, 24 of 24)

- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [Speech Emotion Diarization: Which Emotion Appears When?](https://mlsanity.com/paper/2306.12991/) (2023)
- [Self-Supervised Learning for Speech Enhancement through Synthesis](https://mlsanity.com/paper/2211.02542/) (2022)
- [BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding](https://mlsanity.com/paper/2408.06851/) (2024)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2309.08030/) (2023)
- [Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters](https://mlsanity.com/paper/2401.05111/) (2024)
- [TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2505.06660/) (2025)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement](https://mlsanity.com/paper/2403.01369/) (2024)
- [SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with Representations from Speech Foundation Models](https://mlsanity.com/paper/2406.08445/) (2024)
- [Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancement](https://mlsanity.com/paper/2408.06911/) (2024)
- [Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features](https://mlsanity.com/paper/2502.04722/) (2025)
- [Towards Robust Assessment of Pathological Voices via Combined Low-Level Descriptors and Foundation Model Representations](https://mlsanity.com/paper/2505.21356/) (2025)
- [Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement](https://mlsanity.com/paper/2306.08406/) (2023)
- [An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutions](https://mlsanity.com/paper/2410.23955/) (2024)
- [Magnitude-Phase Dual-Path Speech Enhancement Network based on Self-Supervised Embedding and Perceptual Contrast Stretch Boosting](https://mlsanity.com/paper/2503.21571/) (2025)
- [Joint Learning using Mixture-of-Expert-Based Representation for Speech Enhancement and Robust Emotion Recognition](https://mlsanity.com/paper/2509.08470/) (2025)
- [Masked Autoencoders as Universal Speech Enhancer](https://mlsanity.com/paper/2602.02413/) (2026)
- [Towards Lightweight Adaptation of Speech Enhancement Models in Real-World Environments](https://mlsanity.com/paper/2603.07471/) (2026)
- [UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations](https://mlsanity.com/paper/2604.14606/) (2026)
- [PhASE-Flow: Phonetic-Conditioned Acoustic Flow Matching in SSL Representation Domain for Speech Enhancement](https://mlsanity.com/paper/2606.17806/) (2026)
- [Bridging Self-Supervised Learning and Speech Enhancement: A Wav2Vec2-Conditioned Framework](https://mlsanity.com/paper/2606.22591/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) representation for speech has achieved state-of-the-art (SOTA) performance on several downstream tasks. However, there remains room for improvement in speech enhancement (SE) tasks. In this study, we used a cross-domain feature to solve the problem that SSL embeddings may lack fine-grained information to regenerate speech signals. By integrating the SSL representation and spectrogram, the result can be significantly boosted. We further study the relationship between the noise robustness of SSL representation via clean-noisy distance (CN distance) and the layer importance for SE. Consequently, we found that SSL representations with lower noise robustness are more important. Furthermore, our experiments on the VCTK-DEMAND dataset demonstrated that fine-tuning an SSL representation with an SE model can outperform the SOTA SSL-based SE methods in PESQ, CSIG and COVL without invoking complicated network architectures. In later experiments, the CN distance in SSL embeddings was observed to increase after fine-tuning. These results verify our expectations and may help design SE-related SSL training in the future.