# All Information is Necessary: Integrating Speech Positive and Negative Information by Contrastive Learning for Speech Enhancement

arXiv: https://arxiv.org/abs/2304.13439 · HTML: https://mlsanity.com/paper/2304.13439/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 25 of 25)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022)
- [PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network](https://mlsanity.com/paper/1911.04697/) (2019)
- [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training](https://mlsanity.com/paper/2110.05752/) (2021)
- [Investigating self-supervised learning for speech enhancement and separation](https://mlsanity.com/paper/2203.07960/) (2022)
- [A Fully Convolutional Neural Network for Speech Enhancement](https://mlsanity.com/paper/1609.07132/) (2016)
- [Raw Waveform-based Speech Enhancement by Fully Convolutional Networks](https://mlsanity.com/paper/1703.02205/) (2017)
- [Interactive Speech and Noise Modeling for Speech Enhancement](https://mlsanity.com/paper/2012.09408/) (2020)
- [Magic ELF: Image Deraining Meets Association Learning and Transformer](https://mlsanity.com/paper/2207.10455/) (2022)
- [Speech Enhancement Using Multi-Stage Self-Attentive Temporal Convolutional Networks](https://mlsanity.com/paper/2102.12078/) (2021)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [Improving Dual-Microphone Speech Enhancement by Learning Cross-Channel Features with Multi-Head Attention](https://mlsanity.com/paper/2205.01280/) (2022)
- [U-Former: Improving Monaural Speech Enhancement with Multi-head Self and Cross Attention](https://mlsanity.com/paper/2205.08681/) (2022)
- [GLD-Net: Improving Monaural Speech Enhancement by Learning Global and Local Dependency Features with GLD Block](https://mlsanity.com/paper/2206.14962/) (2022)

## Built on by (most cited first, 1 of 1)

- [SuperCodec: A Neural Speech Codec with Selective Back-Projection Network](https://mlsanity.com/paper/2407.20530/) (2024)

## Abstract (arXiv)

Monaural speech enhancement (SE) is an ill-posed problem due to the irreversible degradation process. Recent methods to achieve SE tasks rely solely on positive information, e.g., ground-truth speech and speech-relevant features. Different from the above, we observe that the negative information, such as original speech mixture and speech-irrelevant features, are valuable to guide the SE model training procedure. In this study, we propose a SE model that integrates both speech positive and negative information for improving SE performance by adopting contrastive learning, in which two innovations have consisted. (1) We design a collaboration module (CM), which contains two parts, contrastive attention for separating relevant and irrelevant features via contrastive learning and interactive attention for establishing the correlation between both speech features in a learnable and self-adaptive manner. (2) We propose a contrastive regularization (CR) built upon contrastive learning to ensure that the estimated speech is pulled closer to the clean speech and pushed far away from the noisy speech in the representation space by integrating self-supervised models. We term the proposed SE network with CM and CR as CMCR-Net. Experimental results demonstrate that our CMCR-Net achieves comparable and superior performance to recent approaches.