# Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders

arXiv: https://arxiv.org/abs/2102.06610 · HTML: https://mlsanity.com/paper/2102.06610/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [A Hybrid DSP/Deep Learning Approach to Real-Time Full-Band Speech Enhancement](https://mlsanity.com/paper/1709.08243/) (2017)
- [Wavenet based low rate speech coding](https://mlsanity.com/paper/1712.01120/) (2017)
- [PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Loss](https://mlsanity.com/paper/2008.04470/) (2020)
- [A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet](https://mlsanity.com/paper/1903.12087/) (2019)
- [Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders](https://mlsanity.com/paper/2008.06892/) (2020)
- [Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding](https://mlsanity.com/paper/2101.00054/) (2020)
- [Towards achieving robust universal neural vocoding](https://mlsanity.com/paper/1811.06292/) (2018)

## Built on by (most cited first, 7 of 7)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Learning to Upsample and Upmix Audio in the Latent Domain](https://mlsanity.com/paper/2506.00681/) (2025)
- [Cross-Scale Vector Quantization for Scalable Neural Speech Coding](https://mlsanity.com/paper/2207.03067/) (2022)
- [Exploratory Evaluation of Speech Content Masking](https://mlsanity.com/paper/2401.03936/) (2024)
- [Cognitive Coding of Speech](https://mlsanity.com/paper/2110.04241/) (2021)
- [Task and Perception-aware Distributed Source Coding for Correlated Speech under Bandwidth-constrained Channels](https://mlsanity.com/paper/2501.17879/) (2025)
- [Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error](https://mlsanity.com/paper/2604.01613/) (2026)

## Abstract (arXiv)

Audio codecs based on discretized neural autoencoders have recently been developed and shown to provide significantly higher compression levels for comparable quality speech output. However, these models are tightly coupled with speech content, and produce unintended outputs in noisy conditions. Based on VQ-VAE autoencoders with WaveRNN decoders, we develop compressor-enhancer encoders and accompanying decoders, and show that they operate well in noisy conditions. We also observe that a compressor-enhancer model performs better on clean speech inputs than a compressor model trained only on clean speech.