mlsanityData to 19 Oct 2026

Investigation of Speech and Noise Latent Representations in Single-channel VAE-based Speech Enhancement

arXiv 2508.05293 ↗First posted 7 Aug 2025eess.AScited by 1

On the map: Speech and audio language models → Speech enhancement and noise suppression

Abstract (arXiv)

Recently, a variational autoencoder (VAE)-based single-channel speech enhancement system using Bayesian permutation training has been proposed, which uses two pretrained VAEs to obtain latent representations for speech and noise. Based on these pretrained VAEs, a noisy VAE learns to generate speech and noise latent representations from noisy speech for speech enhancement. Modifying the pretrained VAE loss terms affects the pretrained speech and noise latent representations. In this paper, we investigate how these different representations affect speech enhancement performance. Experiments on the DNS3, WSJ0-QUT, and VoiceBank-DEMAND datasets show that a latent space where speech and noise representations are clearly separated significantly improves performance over standard VAEs, which produce overlapping speech and noise representations.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models. Builds on 21 of its 29 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

21 of 29 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023A Deep Representation Learning-based Speech Enhancement Method Using Complex Convolution Recurrent Variational Autoencoder15 Dec
  2. 2023Investigating the Design Space of Diffusion Models for Speech Enhancement7 Dec
  3. 2023Unsupervised speech enhancement with diffusion-based generative models19 Sep
  4. 2023Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder19 Sep
  5. 2022Fast and efficient speech enhancement with variational autoencoders2 Nov
  6. 2022A weighted-variance variational autoencoder model for speech enhancement2 Nov
  7. 2022CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement22 Sep
  8. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  9. 2022A deep representation learning speech enhancement method using β-VAE11 May
  10. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  11. 2022A Bayesian Permutation training deep representation learning method for speech enhancement with variational autoencoder24 Jan
  12. 2021Unsupervised Speech Enhancement using Dynamical Variational Auto-Encoders23 Jun
1 more
  1. 2021Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder17 Feb

Other references · 8

  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 316
  2. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 262
  3. 2017Variational Inference of Disentangled Latent Concepts from Unlabeled Observationscited by 244
  4. 2021Interspeech 2021 Deep Noise Suppression Challengecited by 96
  5. 2020Improving GANs for Speech Enhancementcited by 31
  6. 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 30
  7. 2018Scalable Factorized Hierarchical Variational Autoencoder Trainingcited by 7
  8. 2022Disentangled Speech Representation Learning for One-Shot Cross-lingual Voice Conversion Using β-VAEcited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025I-DCCRN-VAE: An Improved Deep Representation Learning Framework for Complex VAE-based Single-channel Speech Enhancementcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Pre-training Feature Guided Diffusion Model for Speech Enhancement
  2. 2022A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training
  3. 2021Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement
  4. 2021Guided Variational Autoencoder for Speech Enhancement With a Supervised Classifier
  5. 2025Efficient Speech Enhancement via Embeddings from Pre-trained Generative Audioencoders
  6. 2023Unsupervised speech enhancement with deep dynamical generative speech and noise models
  7. 2021Incorporating Real-world Noisy Speech in Neural-network-based Speech Enhancement Systems
  8. 2022Audio-visual speech enhancement with a deep Kalman filter generative model