# Interactive Speech and Noise Modeling for Speech Enhancement

arXiv: https://arxiv.org/abs/2012.09408 · HTML: https://mlsanity.com/paper/2012.09408/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2020.

## Built on by (most cited first, 13 of 13)

- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement](https://mlsanity.com/paper/2209.11112/) (2022)
- [Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2110.05267/) (2021)
- [Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2203.14838/) (2022)
- [Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognition](https://mlsanity.com/paper/2302.11362/) (2023)
- [End-to-End Neural Speech Coding for Real-Time Communications](https://mlsanity.com/paper/2201.09429/) (2022)
- [Multi-View Attention Transfer for Efficient Speech Enhancement](https://mlsanity.com/paper/2208.10367/) (2022)
- [DeepVQE: Real Time Deep Voice Quality Enhancement for Joint Acoustic Echo Cancellation, Noise Suppression and Dereverberation](https://mlsanity.com/paper/2306.03177/) (2023)
- [Continuous Speech Separation with Ad Hoc Microphone Arrays](https://mlsanity.com/paper/2103.02378/) (2021)
- [ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression](https://mlsanity.com/paper/2305.16665/) (2023)
- [A Two-stage Complex Network using Cycle-consistent Generative Adversarial Networks for Speech Enhancement](https://mlsanity.com/paper/2109.02011/) (2021)
- [D4AM: A General Denoising Framework for Downstream Acoustic Models](https://mlsanity.com/paper/2311.16595/) (2023)
- [Time-Variance Aware Real-Time Speech Enhancement](https://mlsanity.com/paper/2302.13063/) (2023)

## Abstract (arXiv)

Speech enhancement is challenging because of the diversity of background noise types. Most of the existing methods are focused on modelling the speech rather than the noise. In this paper, we propose a novel idea to model speech and noise simultaneously in a two-branch convolutional neural network, namely SN-Net. In SN-Net, the two branches predict speech and noise, respectively. Instead of information fusion only at the final output layer, interaction modules are introduced at several intermediate feature domains between the two branches to benefit each other. Such an interaction can leverage features learned from one branch to counteract the undesired part and restore the missing component of the other and thus enhance their discrimination capabilities. We also design a feature extraction module, namely residual-convolution-and-attention (RA), to capture the correlations along temporal and frequency dimensions for both the speech and the noises. Evaluations on public datasets show that the interaction module plays a key role in simultaneous modeling and the SN-Net outperforms the state-of-the-art by a large margin on various evaluation metrics. The proposed SN-Net also shows superior performance for speaker separation.