# Investigating self-supervised learning for speech enhancement and separation

arXiv: https://arxiv.org/abs/2203.07960 · HTML: https://mlsanity.com/paper/2203.07960/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2022.

## Built on by (most cited first, 30 of 35)

- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models](https://mlsanity.com/paper/2305.17651/) (2023)
- [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [Audio-Visual Speech Enhancement with Score-Based Generative Models](https://mlsanity.com/paper/2306.01432/) (2023)
- [Adapting Multi-Lingual ASR Models for Handling Multiple Talkers](https://mlsanity.com/paper/2305.18747/) (2023)
- [TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms](https://mlsanity.com/paper/2405.01242/) (2024)
- [Self-Supervised Learning for Speech Enhancement through Synthesis](https://mlsanity.com/paper/2211.02542/) (2022)
- [BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embedding](https://mlsanity.com/paper/2408.06851/) (2024)
- [From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech](https://mlsanity.com/paper/2503.16956/) (2025)
- [MTI-Net: A Multi-Target Speech Intelligibility Prediction Model](https://mlsanity.com/paper/2204.03310/) (2022)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [Speech separation with large-scale self-supervised learning](https://mlsanity.com/paper/2211.05172/) (2022)
- [AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2309.08030/) (2023)
- [PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement](https://mlsanity.com/paper/2511.13300/) (2025)
- [Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation](https://mlsanity.com/paper/2210.12635/) (2022)
- [Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts](https://mlsanity.com/paper/2211.06493/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Explaining Relationships Among Research Papers](https://mlsanity.com/paper/2402.13426/) (2024)
- [StableQuant: Layer Adaptive Post-Training Quantization for Speech Foundation Models](https://mlsanity.com/paper/2504.14915/) (2025)
- [DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective](https://mlsanity.com/paper/2507.02911/) (2025)
- [AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies](https://mlsanity.com/paper/2402.13152/) (2024)
- [A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement](https://mlsanity.com/paper/2403.01369/) (2024)
- [Heterogeneous Space Fusion and Dual-Dimension Attention: A New Paradigm for Speech Enhancement](https://mlsanity.com/paper/2408.06911/) (2024)
- [Speech Separation based on Contrastive Learning and Deep Modularization](https://mlsanity.com/paper/2305.10652/) (2023)
- [Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement](https://mlsanity.com/paper/2306.08406/) (2023)
- [Magnitude-Phase Dual-Path Speech Enhancement Network based on Self-Supervised Embedding and Perceptual Contrast Stretch Boosting](https://mlsanity.com/paper/2503.21571/) (2025)
- [Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation](https://mlsanity.com/paper/2504.02302/) (2025)
- [Masked Autoencoders as Universal Speech Enhancer](https://mlsanity.com/paper/2602.02413/) (2026)

## Abstract (arXiv)

Speech enhancement and separation are two fundamental tasks for robust speech processing. Speech enhancement suppresses background noise while speech separation extracts target speech from interfering speakers. Despite a great number of supervised learning-based enhancement and separation methods having been proposed and achieving good performance, studies on applying self-supervised learning (SSL) to enhancement and separation are limited. In this paper, we evaluate 13 SSL upstream methods on speech enhancement and separation downstream tasks. Our experimental results on Voicebank-DEMAND and Libri2Mix show that some SSL representations consistently outperform baseline features including the short-time Fourier transform (STFT) magnitude and log Mel filterbank (FBANK). Furthermore, we analyze the factors that make existing SSL frameworks difficult to apply to speech enhancement and separation and discuss the representation properties desired for both tasks. Our study is included as the official speech enhancement and separation downstreams for SUPERB.