mlsanityData to 19 Oct 2026

Speech Enhancement with Multi-granularity Vector Quantization

arXiv 2302.08342 ↗First posted 16 Feb 2023eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Speech enhancement and noise suppression

Abstract (arXiv)

With advances in deep learning, neural network based speech enhancement (SE) has developed rapidly in the last decade. Meanwhile, the self-supervised pre-trained model and vector quantization (VQ) have achieved excellent performance on many speech-related tasks, while they are less explored on SE. As it was shown in our previous work that utilizing a VQ module to discretize noisy speech representations is beneficial for speech denoising, in this work we therefore study the impact of using VQ at different layers with different number of codebooks. Different VQ modules indeed enable to extract multiple-granularity speech features. Following an attention mechanism, the contextual features extracted by a pre-trained model are fused with the local features extracted by the encoder, such that both global and local information are preserved to reconstruct the enhanced speech. Experimental results on the Valentini dataset show that the proposed model can improve the SE performance, where the impact of choosing pre-trained models is also revealed.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models. Builds on 24 of its 37 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

24 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Speech Enhancement Using Self-Supervised Pre-Trained Model and Vector Quantization28 Sep
  2. 2022Joint Training of Speech Enhancement and Self-supervised Model for Noise-robust ASR26 May
  3. 2022Boosting Self-Supervised Embeddings for Speech Enhancement7 Apr
  4. 2022Investigating self-supervised learning for speech enhancement and separation15 Mar

Other references · 20

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,797
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,404
  5. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 782
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 774
  7. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 596
  8. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 284
  9. 2020Real Time Speech Enhancement in the Waveform Domaincited by 282
  10. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 262
  11. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 161
  12. 2022VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Featurecited by 54
8 more
  1. 2018Improved Speech Enhancement with the Wave-U-Netcited by 41
  2. 2022Speech Denoising in the Waveform Domain with Self-Attentioncited by 34
  3. 2022A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognitioncited by 32
  4. 2017Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verificationcited by 31
  5. 2020Improving GANs for Speech Enhancementcited by 31
  6. 2022Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learningcited by 25
  7. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoderscited by 20
  8. 2022Cross-Scale Vector Quantization for Scalable Neural Speech Codingcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech
  2. 2022A Two-Stage Deep Representation Learning-Based Speech Enhancement Method Using Variational Autoencoder and Adversarial Training
  3. 2022A Bayesian Permutation training deep representation learning method for speech enhancement with variational autoencoder
  4. 2023Continuous Modeling of the Denoising Process for Speech Enhancement Based on Deep Learning
  5. 2026Towards Robust Generative Speech Enhancement Using Vector Quantisation-Based Neural Audio Codec
  6. 2022A deep representation learning speech enhancement method using β-VAE
  7. 2024Pre-training Feature Guided Diffusion Model for Speech Enhancement
  8. 2025Investigation of Speech and Noise Latent Representations in Single-channel VAE-based Speech Enhancement