# Inference and Denoise: Causal Inference-based Neural Speech Enhancement

arXiv: https://arxiv.org/abs/2211.01189 · HTML: https://mlsanity.com/paper/2211.01189/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Causal Transformer for Estimating Counterfactual Outcomes](https://mlsanity.com/paper/2204.07258/) (2022)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [CMGAN: Conformer-based Metric GAN for Speech Enhancement](https://mlsanity.com/paper/2203.15149/) (2022)
- [On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.07281/) (2020)
- [MANNER: Multi-view Attention Network for Noise Erasure](https://mlsanity.com/paper/2203.02181/) (2022)
- [A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection](https://mlsanity.com/paper/2101.02919/) (2021)
- [Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition](https://mlsanity.com/paper/2112.05820/) (2021)
- [Counterfactually Guided Off-policy Transfer in Clinical Settings](https://mlsanity.com/paper/2006.11654/) (2020)
- [Perceptual Loss based Speech Denoising with an ensemble of Audio Pattern Recognition and Self-Supervised Models](https://mlsanity.com/paper/2010.11860/) (2020)
- [Training a Resilient Q-Network against Observational Interference](https://mlsanity.com/paper/2102.09677/) (2021)
- [Sparse Mixture of Local Experts for Efficient Speech Enhancement](https://mlsanity.com/paper/2005.08128/) (2020)

## Abstract (arXiv)

This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. Based on the potential outcome framework, the proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE. Specifically, we use the presence of noise as guidance for EM selection during training, and the noise detector selects the enhancement module according to the prediction of the presence of noise for each frame. Moreover, we derived a SE-specific average treatment effect to quantify the causal effect adequately. Experimental evidence demonstrates that CISE outperforms a non-causal mask-based SE approach in the studied settings and has better performance and efficiency than more complex SE models.