# A deep representation learning speech enhancement method using β-VAE

arXiv: https://arxiv.org/abs/2205.05581 · HTML: https://mlsanity.com/paper/2205.05581/

## Key facts

- Cited by 4 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 11 May 2022.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 20 of 20)

- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesis](https://mlsanity.com/paper/2110.08813/) (2021)
- [Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization](https://mlsanity.com/paper/1710.11439/) (2017)
- [Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder](https://mlsanity.com/paper/2102.08706/) (2021)
- [A variance modeling framework based on variational autoencoders for speech enhancement](https://mlsanity.com/paper/1902.01605/) (2019)
- [On The Compensation Between Magnitude and Phase in Speech Separation](https://mlsanity.com/paper/2108.05470/) (2021)
- [ICASSP 2021 Deep Noise Suppression Challenge: Decoupling Magnitude and Phase Optimization with a Two-Stage Deep Network](https://mlsanity.com/paper/2102.04198/) (2021)
- [Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization](https://mlsanity.com/paper/1811.06713/) (2018)
- [Guided Variational Autoencoder for Speech Enhancement With a Supervised Classifier](https://mlsanity.com/paper/2102.06454/) (2021)
- [Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement](https://mlsanity.com/paper/2106.02896/) (2021)
- [Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2105.08970/) (2021)
- [Disentangled Speech Representation Learning Based on Factorized Hierarchical Variational Autoencoder with Self-Supervised Objective](https://mlsanity.com/paper/2204.02166/) (2022)
- [A Bayesian Permutation training deep representation learning method for speech enhancement with variational autoencoder](https://mlsanity.com/paper/2201.09875/) (2022)
- [Model-based Speech Enhancement for Intelligibility Improvement in Binaural Hearing Aids](https://mlsanity.com/paper/1806.04885/) (2018)

## Built on by (most cited first, 4 of 4)

- [A Deep Representation Learning-based Speech Enhancement Method Using Complex Convolution Recurrent Variational Autoencoder](https://mlsanity.com/paper/2312.09620/) (2023)
- [A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training](https://mlsanity.com/paper/2211.09166/) (2022)
- [Investigation of Speech and Noise Latent Representations in Single-channel VAE-based Speech Enhancement](https://mlsanity.com/paper/2508.05293/) (2025)
- [I-DCCRN-VAE: An Improved Deep Representation Learning Framework for Complex VAE-based Single-channel Speech Enhancement](https://mlsanity.com/paper/2510.12485/) (2025)

## Abstract (arXiv)

In previous work, we proposed a variational autoencoder-based (VAE) Bayesian permutation training speech enhancement (SE) method (PVAE) which indicated that the SE performance of the traditional deep neural network-based (DNN) method could be improved by deep representation learning (DRL). Based on our previous work, we in this paper propose to use β-VAE to further improve PVAE's ability of representation learning. More specifically, our β-VAE can improve PVAE's capacity of disentangling different latent variables from the observed signal without the trade-off problem between disentanglement and signal reconstruction. This trade-off problem widely exists in previous β-VAE algorithms. Unlike the previous β-VAE algorithms, the proposed β-VAE strategy can also be used to optimize the DNN's structure. This means that the proposed method can not only improve PVAE's SE performance but also reduce the number of PVAE training parameters. The experimental results show that the proposed method can acquire better speech and noise latent representation than PVAE. Meanwhile, it also obtains a higher scale-invariant signal-to-distortion ratio, speech quality, and speech intelligibility.