# Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder

arXiv: https://arxiv.org/abs/1910.06464 · HTML: https://mlsanity.com/paper/1910.06464/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2019.

## Built on by (most cited first, 30 of 49)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [VampNet: Music Generation via Masked Acoustic Token Modeling](https://mlsanity.com/paper/2307.04686/) (2023)
- [SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound](https://mlsanity.com/paper/2405.00233/) (2024)
- [SNAC: Multi-Scale Neural Audio Codec](https://mlsanity.com/paper/2410.14411/) (2024)
- [JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models](https://mlsanity.com/paper/2308.04729/) (2023)
- [RepCodec: A Speech Representation Codec for Speech Tokenization](https://mlsanity.com/paper/2309.00169/) (2023)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation](https://mlsanity.com/paper/2310.19180/) (2023)
- [EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis](https://mlsanity.com/paper/2311.08667/) (2023)
- [Neural Distributed Source Coding](https://mlsanity.com/paper/2106.02797/) (2021)
- [Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference](https://mlsanity.com/paper/2409.12117/) (2024)
- [Disentangled Feature Learning for Real-Time Neural Speech Coding](https://mlsanity.com/paper/2211.11960/) (2022)
- [A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS](https://mlsanity.com/paper/2209.10887/) (2022)
- [Generative De-Quantization for Neural Speech Codec via Latent Diffusion](https://mlsanity.com/paper/2311.08330/) (2023)
- [Enhancing into the codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders](https://mlsanity.com/paper/2102.06610/) (2021)
- [HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding](https://mlsanity.com/paper/2107.10843/) (2021)
- [End-to-End Neural Speech Coding for Real-Time Communications](https://mlsanity.com/paper/2201.09429/) (2022)
- [Latent-Domain Predictive Neural Speech Coding](https://mlsanity.com/paper/2207.08363/) (2022)
- [Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models](https://mlsanity.com/paper/2409.12139/) (2024)
- [Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models](https://mlsanity.com/paper/2110.14182/) (2021)
- [Disentangling speech from surroundings with neural embeddings](https://mlsanity.com/paper/2203.15578/) (2022)
- [Learning Source Disentanglement in Neural Audio Codec](https://mlsanity.com/paper/2409.11228/) (2024)
- [Scalable and Efficient Neural Speech Coding: A Hybrid Design](https://mlsanity.com/paper/2103.14776/) (2021)
- [Ultra-Low-Bitrate Speech Coding with Pretrained Transformers](https://mlsanity.com/paper/2207.02262/) (2022)
- [Gull: A Generative Multifunctional Audio Codec](https://mlsanity.com/paper/2404.04947/) (2024)
- [SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec](https://mlsanity.com/paper/2508.02849/) (2025)
- [End-to-End Binaural Speech Synthesis](https://mlsanity.com/paper/2207.03697/) (2022)

## Abstract (arXiv)

In order to efficiently transmit and store speech signals, speech codecs create a minimally redundant representation of the input signal which is then decoded at the receiver with the best possible perceptual quality. In this work we demonstrate that a neural network architecture based on VQ-VAE with a WaveNet decoder can be used to perform very low bit-rate speech coding with high reconstruction quality. A prosody-transparent and speaker-independent model trained on the LibriSpeech corpus coding audio at 1.6 kbps exhibits perceptual quality which is around halfway between the MELP codec at 2.4 kbps and AMR-WB codec at 23.05 kbps. In addition, when training on high-quality recorded speech with the test speaker included in the training set, a model coding speech at 1.6 kbps produces output of similar perceptual quality to that generated by AMR-WB at 23.05 kbps.