# NESC: Robust Neural End-2-End Speech Coding with GANs

arXiv: https://arxiv.org/abs/2207.03282 · HTML: https://mlsanity.com/paper/2207.03282/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 7 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric](https://mlsanity.com/paper/2004.09584/) (2020)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [Wavenet based low rate speech coding](https://mlsanity.com/paper/1712.01120/) (2017)
- [End-to-End Optimized Speech Coding with Deep Neural Networks](https://mlsanity.com/paper/1710.09064/) (2017)
- [Generative Speech Coding with Predictive Variance Regularization](https://mlsanity.com/paper/2102.09660/) (2021)
- [A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet](https://mlsanity.com/paper/1903.12087/) (2019)
- [Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding](https://mlsanity.com/paper/1906.07769/) (2019)
- [Audio Codec Enhancement with Generative Adversarial Networks](https://mlsanity.com/paper/2001.09653/) (2020)
- [Improving Opus Low Bit Rate Quality with Neural Speech Synthesis](https://mlsanity.com/paper/1905.04628/) (2019)
- [Efficient And Scalable Neural Residual Waveform Coding With Collaborative Quantization](https://mlsanity.com/paper/2002.05604/) (2020)
- [A Streamwise GAN Vocoder for Wideband Speech Coding at Very Low Bit Rate](https://mlsanity.com/paper/2108.04051/) (2021)
- [Convolutional Neural Networks to Enhance Coded Speech](https://mlsanity.com/paper/1806.09411/) (2018)
- [PostGAN: A GAN-Based Post-Processor to Enhance the Quality of Coded Speech](https://mlsanity.com/paper/2201.13093/) (2022)
- [Enhancement Of Coded Speech Using a Mask-Based Post-Filter](https://mlsanity.com/paper/2010.05571/) (2020)

## Built on by (most cited first, 3 of 3)

- [FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Rates](https://mlsanity.com/paper/2409.17635/) (2024)
- [Framewise WaveGAN: High Speed Adversarial Vocoder in Time Domain with Very Low Computational Complexity](https://mlsanity.com/paper/2212.04532/) (2022)
- [On the Design of Diffusion-based Neural Speech Codecs](https://mlsanity.com/paper/2504.08470/) (2025)

## Abstract (arXiv)

Neural networks have proven to be a formidable tool to tackle the problem of speech coding at very low bit rates. However, the design of a neural coder that can be operated robustly under real-world conditions remains a major challenge. Therefore, we present Neural End-2-End Speech Codec (NESC) a robust, scalable end-to-end neural speech codec for high-quality wideband speech coding at 3 kbps. The encoder uses a new architecture configuration, which relies on our proposed Dual-PathConvRNN (DPCRNN) layer, while the decoder architecture is based on our previous work Streamwise-StyleMelGAN. Our subjective listening tests on clean and noisy speech show that NESC is particularly robust to unseen conditions and signal perturbations.