# Unsupervised speech enhancement with deep dynamical generative speech and noise models

arXiv: https://arxiv.org/abs/2306.07820 · HTML: https://mlsanity.com/paper/2306.07820/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Conditional Diffusion Probabilistic Model for Speech Enhancement](https://mlsanity.com/paper/2202.05256/) (2022)
- [Dynamical Variational Autoencoders: A Comprehensive Review](https://mlsanity.com/paper/2008.12595/) (2020)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [A Recurrent Variational Autoencoder for Speech Enhancement](https://mlsanity.com/paper/1910.10942/) (2019)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [A variance modeling framework based on variational autoencoders for speech enhancement](https://mlsanity.com/paper/1902.01605/) (2019)
- [Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders](https://mlsanity.com/paper/2106.12271/) (2021)
- [MetricGAN-U: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech](https://mlsanity.com/paper/2110.05866/) (2021)
- [Speech Denoising Without Clean Training Data: A Noise2Noise Approach](https://mlsanity.com/paper/2104.03838/) (2021)
- [Noisy-target Training: A Training Strategy for DNN-based Speech Enhancement without Clean Speech](https://mlsanity.com/paper/2101.08625/) (2021)
- [Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization](https://mlsanity.com/paper/1811.06713/) (2018)
- [Guided Variational Autoencoder for Speech Enhancement With a Supervised Classifier](https://mlsanity.com/paper/2102.06454/) (2021)
- [Speech enhancement with variational autoencoders and alpha-stable distributions](https://mlsanity.com/paper/1902.03926/) (2019)
- [CycleGAN-based Non-parallel Speech Enhancement with an Adaptive Attention-in-attention Mechanism](https://mlsanity.com/paper/2107.13143/) (2021)
- [Improving Deep Speech Denoising by Noisy2Noisy Signal Mapping](https://mlsanity.com/paper/1904.12069/) (2019)
- [A Statistically Principled and Computationally Efficient Approach to Speech Enhancement using Variational Autoencoders](https://mlsanity.com/paper/1905.01209/) (2019)
- [A Benchmark of Dynamical Variational Autoencoders applied to Speech Spectrogram Modeling](https://mlsanity.com/paper/2106.06500/) (2021)
- [Fast and efficient speech enhancement with variational autoencoders](https://mlsanity.com/paper/2211.02728/) (2022)
- [A Bayesian Permutation training deep representation learning method for speech enhancement with variational autoencoder](https://mlsanity.com/paper/2201.09875/) (2022)

## Built on by (most cited first, 1 of 1)

- [Diffusion-based Unsupervised Audio-visual Speech Enhancement](https://mlsanity.com/paper/2410.05301/) (2024)

## Abstract (arXiv)

This work builds on a previous work on unsupervised speech enhancement using a dynamical variational autoencoder (DVAE) as the clean speech model and non-negative matrix factorization (NMF) as the noise model. We propose to replace the NMF noise model with a deep dynamical generative model (DDGM) depending either on the DVAE latent variables, or on the noisy observations, or on both. This DDGM can be trained in three configurations: noise-agnostic, noise-dependent and noise adaptation after noise-dependent training. Experimental results show that the proposed method achieves competitive performance compared to state-of-the-art unsupervised speech enhancement methods, while the noise-dependent training configuration yields a much more time-efficient inference process.