# Feature Learning and Ensemble Pre-Tasks Based Self-Supervised Speech Denoising and Dereverberation

arXiv: https://arxiv.org/abs/2206.04962 · HTML: https://mlsanity.com/paper/2206.04962/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 17 of 17)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [Self-supervised Learning for Speech Enhancement](https://mlsanity.com/paper/2006.10388/) (2020)
- [Progressive Joint Modeling in Unsupervised Single-channel Overlapped Speech Recognition](https://mlsanity.com/paper/1707.07048/) (2017)
- [Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement](https://mlsanity.com/paper/2112.05036/) (2021)
- [Stabilizing Label Assignment for Speech Separation by Self-supervised Pre-training](https://mlsanity.com/paper/2010.15366/) (2020)
- [Self-Supervised Learning from Contrastive Mixtures for Personalized Speech Enhancement](https://mlsanity.com/paper/2011.03426/) (2020)
- [Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning](https://mlsanity.com/paper/2110.09930/) (2021)
- [Complex ratio masking for singing voice separation](https://mlsanity.com/paper/2011.02008/) (2020)
- [Improving Distantly Supervised Relation Extraction with Self-Ensemble Noise Filtering](https://mlsanity.com/paper/2108.09689/) (2021)

## Abstract (arXiv)

Self-supervised learning (SSL) achieves great success in monaural speech enhancement, while the accuracy of the target speech estimation, particularly for unseen speakers, remains inadequate with existing pre-tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, and spoken content, the latent representation for speech enhancement becomes a tough task. In this paper, we study the effectiveness of each feature which is commonly used in speech enhancement and exploit the feature combination in the SSL case. Besides, we propose an ensemble training strategy. The latent representation of the clean speech signal is learned, meanwhile, the dereverberated mask and the estimated ratio mask are exploited to denoise and dereverberate the mixture. The latent representation learning and the masks estimation are considered as two pre-tasks in the training stage. In addition, to study the effectiveness between the pre-tasks, we compare different training routines to train the model and further refine the performance. The NOISEX and DAPS corpora are used to evaluate the efficacy of the proposed method, which also outperforms the state-of-the-art methods.