# End-to-End Optimized Speech Coding with Deep Neural Networks

arXiv: https://arxiv.org/abs/1710.09064 · HTML: https://mlsanity.com/paper/1710.09064/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2017.

## Built on by (most cited first, 22 of 22)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [SNAC: Multi-Scale Neural Audio Codec](https://mlsanity.com/paper/2410.14411/) (2024)
- [RepCodec: A Speech Representation Codec for Speech Tokenization](https://mlsanity.com/paper/2309.00169/) (2023)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation](https://mlsanity.com/paper/2405.13762/) (2024)
- [FlowDec: A flow-based full-band general audio codec with high perceptual quality](https://mlsanity.com/paper/2503.01485/) (2025)
- [Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference](https://mlsanity.com/paper/2409.12117/) (2024)
- [HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding](https://mlsanity.com/paper/2107.10843/) (2021)
- [Disentangling speech from surroundings with neural embeddings](https://mlsanity.com/paper/2203.15578/) (2022)
- [Learning Source Disentanglement in Neural Audio Codec](https://mlsanity.com/paper/2409.11228/) (2024)
- [Scalable and Efficient Neural Speech Coding: A Hybrid Design](https://mlsanity.com/paper/2103.14776/) (2021)
- [Gull: A Generative Multifunctional Audio Codec](https://mlsanity.com/paper/2404.04947/) (2024)
- [Cross-Scale Vector Quantization for Scalable Neural Speech Coding](https://mlsanity.com/paper/2207.03067/) (2022)
- [NESC: Robust Neural End-2-End Speech Coding with GANs](https://mlsanity.com/paper/2207.03282/) (2022)
- [FlowMAC: Conditional Flow Matching for Audio Coding at Low Bit Rates](https://mlsanity.com/paper/2409.17635/) (2024)
- [Neural Speech and Audio Coding: Modern AI Technology Meets Traditional Codecs](https://mlsanity.com/paper/2408.06954/) (2024)
- [Deep Neural Networks and End-to-End Learning for Audio Compression](https://mlsanity.com/paper/2105.11681/) (2021)
- [Conditional variational autoencoder to improve neural audio synthesis for polyphonic music sound](https://mlsanity.com/paper/2211.08715/) (2022)
- [Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization](https://mlsanity.com/paper/2109.11168/) (2021)
- [EditGen: Harnessing Cross-Attention Control for Instruction-Based Auto-Regressive Audio Editing](https://mlsanity.com/paper/2507.11096/) (2025)

## Abstract (arXiv)

Modern compression algorithms are often the result of laborious domain-specific research; industry standards such as MP3, JPEG, and AMR-WB took years to develop and were largely hand-designed. We present a deep neural network model which optimizes all the steps of a wideband speech coding pipeline (compression, quantization, entropy coding, and decompression) end-to-end directly from raw speech data -- no manual feature engineering necessary, and it trains in hours. In testing, our DNN-based coder performs on par with the AMR-WB standard at a variety of bitrates (~9kbps up to ~24kbps). It also runs in realtime on a 3.8GhZ Intel CPU.