# A Recurrent Variational Autoencoder for Speech Enhancement

arXiv: https://arxiv.org/abs/1910.10942 · HTML: https://mlsanity.com/paper/1910.10942/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2019.

## Built on by (most cited first, 30 of 38)

- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI](https://mlsanity.com/paper/2303.13336/) (2023)
- [A Study on Speech Enhancement Based on Diffusion Probabilistic Model](https://mlsanity.com/paper/2107.11876/) (2021)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting](https://mlsanity.com/paper/2307.08243/) (2023)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos](https://mlsanity.com/paper/2409.02638/) (2024)
- [SRTNet: Time Domain Speech Enhancement Via Stochastic Refinement](https://mlsanity.com/paper/2210.16805/) (2022)
- [Unsupervised speech enhancement with diffusion-based generative models](https://mlsanity.com/paper/2309.10450/) (2023)
- [SE-Bridge: Speech Enhancement with Consistent Brownian Bridge](https://mlsanity.com/paper/2305.13796/) (2023)
- [Diffusion-based speech enhancement with a weighted generative-supervised learning loss](https://mlsanity.com/paper/2309.10457/) (2023)
- [Neural Blind Source Separation and Diarization for Distant Speech Recognition](https://mlsanity.com/paper/2406.08396/) (2024)
- [Deep neural network techniques for monaural speech enhancement: state of the art analysis](https://mlsanity.com/paper/2212.00369/) (2022)
- [Learning Robust Latent Representations for Controllable Speech Synthesis](https://mlsanity.com/paper/2105.04458/) (2021)
- [GANtron: Emotional Speech Synthesis with Generative Adversarial Networks](https://mlsanity.com/paper/2110.03390/) (2021)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [Audio-visual speech enhancement with a deep Kalman filter generative model](https://mlsanity.com/paper/2211.00988/) (2022)
- [Fast and efficient speech enhancement with variational autoencoders](https://mlsanity.com/paper/2211.02728/) (2022)
- [Diff-SV: A Unified Hierarchical Framework for Noise-Robust Speaker Verification Using Score-Based Diffusion Probabilistic Models](https://mlsanity.com/paper/2309.08320/) (2023)
- [HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality](https://mlsanity.com/paper/2410.16430/) (2024)
- [A Hybrid Model for Weakly-Supervised Speech Dereverberation](https://mlsanity.com/paper/2502.06839/) (2025)
- [Plasma State Monitoring and Disruption Characterization using Multimodal VAEs](https://mlsanity.com/paper/2504.17710/) (2025)
- [Speech Modeling with a Hierarchical Transformer Dynamical VAE](https://mlsanity.com/paper/2303.09404/) (2023)
- [Unsupervised speech enhancement with deep dynamical generative speech and noise models](https://mlsanity.com/paper/2306.07820/) (2023)
- [On the Impact of Sampling on Deep Sequential State Estimation](https://mlsanity.com/paper/2311.17006/) (2023)
- [Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation](https://mlsanity.com/paper/2312.04167/) (2023)
- [Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement](https://mlsanity.com/paper/2508.20859/) (2025)
- [Unsupervised Multiple-Object Tracking with a Dynamical Variational Autoencoder](https://mlsanity.com/paper/2202.09315/) (2022)
- [A weighted-variance variational autoencoder model for speech enhancement](https://mlsanity.com/paper/2211.00990/) (2022)
- [Integrating Uncertainty into Neural Network-based Speech Enhancement](https://mlsanity.com/paper/2305.08744/) (2023)

## Abstract (arXiv)

This paper presents a generative approach to speech enhancement based on a recurrent variational autoencoder (RVAE). The deep generative speech model is trained using clean speech signals only, and it is combined with a nonnegative matrix factorization noise model for speech enhancement. We propose a variational expectation-maximization algorithm where the encoder of the RVAE is fine-tuned at test time, to approximate the distribution of the latent variables given the noisy speech observations. Compared with previous approaches based on feed-forward fully-connected architectures, the proposed recurrent deep generative speech model induces a posterior temporal dynamic over the latent variables, which is shown to improve the speech enhancement results.