# Leveraging Heteroscedastic Uncertainty in Learning Complex Spectral Mapping for Single-channel Speech Enhancement

arXiv: https://arxiv.org/abs/2211.08624 · HTML: https://mlsanity.com/paper/2211.08624/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks](https://mlsanity.com/paper/2203.09168/) (2022)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement](https://mlsanity.com/paper/1909.01019/) (2019)
- [Complex spectrogram enhancement by convolutional neural network with multi-metrics learning](https://mlsanity.com/paper/1704.08504/) (2017)
- [Speech Quality Assessment through MOS using Non-Matching References](https://mlsanity.com/paper/2206.12285/) (2022)
- [Taylor, Can You Hear Me Now? A Taylor-Unfolding Framework for Monaural Speech Enhancement](https://mlsanity.com/paper/2205.00206/) (2022)
- [Integrating Statistical Uncertainty into Neural Network-Based Speech Enhancement](https://mlsanity.com/paper/2203.02288/) (2022)
- [Learning-based personal speech enhancement for teleconferencing by exploiting spatial-spectral features](https://mlsanity.com/paper/2112.05686/) (2021)

## Built on by (most cited first, 1 of 1)

- [Uncertainty-Driven Radar-Inertial Fusion for Instantaneous 3D Ego-Velocity Estimation](https://mlsanity.com/paper/2506.14294/) (2025)

## Abstract (arXiv)

Most speech enhancement (SE) models learn a point estimate and do not make use of uncertainty estimation in the learning process. In this paper, we show that modeling heteroscedastic uncertainty by minimizing a multivariate Gaussian negative log-likelihood (NLL) improves SE performance at no extra cost. During training, our approach augments a model learning complex spectral mapping with a temporary submodel to predict the covariance of the enhancement error at each time-frequency bin. Due to unrestricted heteroscedastic uncertainty, the covariance introduces an undersampling effect, detrimental to SE performance. To mitigate undersampling, our approach inflates the uncertainty lower bound and weights each loss component with their uncertainty, effectively compensating severely undersampled components with more penalties. Our multivariate setting reveals common covariance assumptions such as scalar and diagonal matrices. By weakening these assumptions, we show that the NLL achieves superior performance compared to popular loss functions including the mean squared error (MSE), mean absolute error (MAE), and scale-invariant signal-to-distortion ratio (SI-SDR).