# A variance modeling framework based on variational autoencoders for speech enhancement

arXiv: https://arxiv.org/abs/1902.01605 · HTML: https://mlsanity.com/paper/1902.01605/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2019.

## Built on by (most cited first, 30 of 33)

- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder](https://mlsanity.com/paper/2102.08706/) (2021)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [Guided Variational Autoencoder for Speech Enhancement With a Supervised Classifier](https://mlsanity.com/paper/2102.06454/) (2021)
- [SRTNet: Time Domain Speech Enhancement Via Stochastic Refinement](https://mlsanity.com/paper/2210.16805/) (2022)
- [Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders](https://mlsanity.com/paper/2305.10734/) (2023)
- [Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2105.08970/) (2021)
- [SE-Bridge: Speech Enhancement with Consistent Brownian Bridge](https://mlsanity.com/paper/2305.13796/) (2023)
- [Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement](https://mlsanity.com/paper/2102.04144/) (2021)
- [Integrating Statistical Uncertainty into Neural Network-Based Speech Enhancement](https://mlsanity.com/paper/2203.02288/) (2022)
- [Expression-preserving face frontalization improves visually assisted speech processing](https://mlsanity.com/paper/2204.02810/) (2022)
- [The impact of removing head movements on audio-visual speech enhancement](https://mlsanity.com/paper/2202.00538/) (2022)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)
- [A Sparsity-promoting Dictionary Model for Variational Autoencoders](https://mlsanity.com/paper/2203.15758/) (2022)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [Audio-visual speech enhancement with a deep Kalman filter generative model](https://mlsanity.com/paper/2211.00988/) (2022)
- [Fast and efficient speech enhancement with variational autoencoders](https://mlsanity.com/paper/2211.02728/) (2022)
- [Sound Source Separation Using Latent Variational Block-Wise Disentanglement](https://mlsanity.com/paper/2402.06683/) (2024)
- [Deep Generative Modeling for Identification of Noisy, Non-Stationary Dynamical Systems](https://mlsanity.com/paper/2410.02079/) (2024)
- [Partially Adaptive Multichannel Joint Reduction of Ego-noise and Environmental Noise](https://mlsanity.com/paper/2303.15042/) (2023)
- [Unsupervised speech enhancement with deep dynamical generative speech and noise models](https://mlsanity.com/paper/2306.07820/) (2023)
- [Selective State Space Model for Monaural Speech Enhancement](https://mlsanity.com/paper/2411.06217/) (2024)
- [A weighted-variance variational autoencoder model for speech enhancement](https://mlsanity.com/paper/2211.00990/) (2022)
- [Integrating Uncertainty into Neural Network-based Speech Enhancement](https://mlsanity.com/paper/2305.08744/) (2023)
- [Self-supervised learning with diffusion-based multichannel speech enhancement for speaker verification under noisy conditions](https://mlsanity.com/paper/2307.02244/) (2023)
- [Combining Deterministic Enhanced Conditions with Dual-Streaming Encoding for Diffusion-Based Speech Enhancement](https://mlsanity.com/paper/2505.13983/) (2025)
- [SEED: Speaker Embedding Enhancement Diffusion Model](https://mlsanity.com/paper/2505.16798/) (2025)

## Abstract (arXiv)

In this paper we address the problem of enhancing speech signals in noisy mixtures using a source separation approach. We explore the use of neural networks as an alternative to a popular speech variance model based on supervised non-negative matrix factorization (NMF). More precisely, we use a variational autoencoder as a speaker-independent supervised generative speech model, highlighting the conceptual similarities that this approach shares with its NMF-based counterpart. In order to be free of generalization issues regarding the noisy recording environments, we follow the approach of having a supervised model only for the target speech signal, the noise model being based on unsupervised NMF. We develop a Monte Carlo expectation-maximization algorithm for inferring the latent variables in the variational autoencoder and estimating the unsupervised model parameters. Experiments show that the proposed method outperforms a semi-supervised NMF baseline and a state-of-the-art fully supervised deep learning approach.