# Self-Supervised Learning based Monaural Speech Enhancement with Complex-Cycle-Consistent

arXiv: https://arxiv.org/abs/2112.11142 · HTML: https://mlsanity.com/paper/2112.11142/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 24 of 24)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation](https://mlsanity.com/paper/2007.13975/) (2020)
- [Guided Variational Autoencoder for Disentanglement Learning](https://mlsanity.com/paper/2004.01255/) (2020)
- [An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04590/) (2021)
- [Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation](https://mlsanity.com/paper/2109.05418/) (2021)
- [PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss](https://mlsanity.com/paper/2008.04470/) (2020)
- [On The Compensation Between Magnitude and Phase in Speech Separation](https://mlsanity.com/paper/2108.05470/) (2021)
- [VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/2007.15256/) (2020)
- [CR-GAN: Learning Complete Representations for Multi-view Generation](https://mlsanity.com/paper/1806.11191/) (2018)
- [Self-supervised Learning for Speech Enhancement](https://mlsanity.com/paper/2006.10388/) (2020)
- [Ultra-Lightweight Speech Separation via Group Communication](https://mlsanity.com/paper/2011.08397/) (2020)
- [Cycle-Consistent Speech Enhancement](https://mlsanity.com/paper/1809.02253/) (2018)
- [Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement](https://mlsanity.com/paper/2112.05036/) (2021)
- [Stabilizing Label Assignment for Speech Separation by Self-supervised Pre-training](https://mlsanity.com/paper/2010.15366/) (2020)
- [Speech enhancement with weakly labelled data from AudioSet](https://mlsanity.com/paper/2102.09971/) (2021)
- [Personalized Speech Enhancement through Self-Supervised Data Augmentation and Purification](https://mlsanity.com/paper/2104.02018/) (2021)
- [Self-Supervised Learning from Contrastive Mixtures for Personalized Speech Enhancement](https://mlsanity.com/paper/2011.03426/) (2020)
- [A Two-stage Complex Network using Cycle-consistent Generative Adversarial Networks for Speech Enhancement](https://mlsanity.com/paper/2109.02011/) (2021)
- [Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning](https://mlsanity.com/paper/2110.09930/) (2021)
- [A Multi-Resolution Front-End for End-to-End Speech Anti-Spoofing](https://mlsanity.com/paper/2110.05087/) (2021)

## Abstract (arXiv)

Recently, self-supervised learning (SSL) techniques have been introduced to solve the monaural speech enhancement problem. Due to the lack of using clean phase information, the enhancement performance is limited in most SSL methods. Therefore, in this paper, we propose a phase-aware self-supervised learning based monaural speech enhancement method. The latent representations of both amplitude and phase are studied in two decoders of the foundation autoencoder (FAE) with only a limited set of clean speech signals independently. Then, the downstream autoencoder (DAE) learns a shared latent space between the clean speech and mixture representations with a large number of unseen mixtures. A complex-cycle-consistent (CCC) mechanism is proposed to minimize the reconstruction loss between the amplitude and phase domains. Besides, it is noticed that if the speech features are extracted as the multi-resolution spectra, the desired information distributed in spectra of different scales can be studied to further boost the performance. The NOISEX and DAPS corpora are used to generate mixtures with different interferences to evaluate the efficacy of the proposed method. It is highlighted that the clean speech and mixtures fed in FAE and DAE are not paired. Both ablation and comparison experimental results show that the proposed method clearly outperforms the state-of-the-art approaches.