A Neural Speech Codec for Noise Robust Speech Coding
On the map: Speech and audio language models → Vocoders and speech synthesis
Abstract (arXiv)
This paper considers the joint compression and enhancement problem for speech signal in the presence of noise. Recently, the SoundStream codec, which relies on end-to-end joint training of an encoder-decoder pair and a residual vector quantizer by a combination of adversarial and reconstruction losses,has shown very promising performance, especially in subjective perception quality. In this work, we provide a theoretical result to show that, to simultaneously achieve low distortion and high perception in the presence of noise, there exist an optimal two-stage optimization procedure for the joint compression and enhancement problem. This procedure firstly optimizes an encoder-decoder pair using only distortion loss and then fixes the encoder to optimize a perceptual decoder using perception loss. Based on this result, we construct a two-stage training framework for joint compression and enhancement of noisy speech signal. Unlike existing training methods which are heuristic, the proposed two-stage training method has a theoretical foundation. Finally, experimental results for various noise and bit-rate conditions are provided. The results demonstrate that a codec trained by the proposed framework can outperform SoundStream and other representative codecs in terms of both objective and subjective evaluation metrics. Code is available at textithttps://github.com/jscscloris/SEStream.
Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 38 of its 60 references that are in the map. First posted on arXiv 8 Sep 2023.
Builds on
38 of 60 references.
Closest ancestors · 2
Same direction, newest first
- 2022End-to-End Neural Speech Coding for Real-Time Communications24 Jan
- 2021SoundStream: An End-to-End Neural Audio Codec7 Jul
Other references · 36
- 2017Neural Discrete Representation Learningcited by 4,843
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,420
- 2016WaveNet: A Generative Model for Raw Audiocited by 1,880
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
- 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,404
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
- 2017The Perception-Distortion Tradeoffcited by 599
- 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 436
- 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 425
- 2018Efficient Neural Audio Synthesiscited by 280
- 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 262
- 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 240
24 more
- 2018Adversarial Audio Synthesiscited by 234
- 2019Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoffcited by 216
- 2022Conditional Diffusion Probabilistic Model for Speech Enhancementcited by 182
- 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 161
- 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 160
- 2017A Wavenet for Speech Denoisingcited by 69
- 2019Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decodercited by 66
- 2017Wavenet based low rate speech codingcited by 51
- 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 46
- 2017End-to-End Waveform Utterance Enhancement for Direct Evaluation Metrics Optimization by Fully Convolutional Neural Networkscited by 43
- 2021Generative Speech Coding with Predictive Variance Regularizationcited by 30
- 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 30
- 2017Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognitioncited by 29
- 2020Real-time Speech Frequency Bandwidth Extensioncited by 28
- 2021On Perceptual Lossy Compression: The Cost of Perceptual Reconstruction and An Optimal Training Frameworkcited by 28
- 2018High-quality speech coding with SampleRNNcited by 23
- 2019Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Codingcited by 21
- 2020Audio Codec Enhancement with Generative Adversarial Networkscited by 17
- 2018DNN-based Source Enhancement to Increase Objective Sound Quality Assessment Scorecited by 16
- 2020Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigmcited by 16
- 2019Improving Opus Low Bit Rate Quality with Neural Speech Synthesiscited by 14
- 2022Optimally Controllable Perceptual Lossy Compressioncited by 14
- 2021Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoderscited by 12
- 2020Source coding of audio signals with a generative modelcited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Ultra-Low-Bitrate Speech Coding with Pretrained Transformers
- 2024Optimizing Neural Speech Codec for Low-Bitrate Compression via Multi-Scale Encoding
- 2025Enhancing Noise Robustness for Neural Speech Codecs through Resource-Efficient Progressive Quantization Perturbation Simulation
- 2025Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates
- 2024APCodec+: A Spectrum-Coding-Based High-Fidelity and High-Compression-Rate Neural Audio Codec with Staged Training Paradigm
- 2022NESC: Robust Neural End-2-End Speech Coding with GANs
- 2026VoCodec: An Efficient Lightweight Low-Bitrate Speech Codec
- 2024ESC: Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers