# Neural Speech and Audio Coding: Modern AI Technology Meets Traditional Codecs

arXiv: https://arxiv.org/abs/2408.06954 · HTML: https://mlsanity.com/paper/2408.06954/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 20 of 20)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [Wavenet based low rate speech coding](https://mlsanity.com/paper/1712.01120/) (2017)
- [End-to-End Optimized Speech Coding with Deep Neural Networks](https://mlsanity.com/paper/1710.09064/) (2017)
- [A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet](https://mlsanity.com/paper/1903.12087/) (2019)
- [Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding](https://mlsanity.com/paper/1906.07769/) (2019)
- [Audio Codec Enhancement with Generative Adversarial Networks](https://mlsanity.com/paper/2001.09653/) (2020)
- [Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Coding](https://mlsanity.com/paper/2101.00054/) (2020)
- [Latent-Domain Predictive Neural Speech Coding](https://mlsanity.com/paper/2207.08363/) (2022)
- [Scalable and Efficient Neural Speech Coding: A Hybrid Design](https://mlsanity.com/paper/2103.14776/) (2021)
- [Convolutional Neural Networks to Enhance Coded Speech](https://mlsanity.com/paper/1806.09411/) (2018)
- [PostGAN: A GAN-Based Post-Processor to Enhance the Quality of Coded Speech](https://mlsanity.com/paper/2201.13093/) (2022)
- [Neural Feature Predictor and Discriminative Residual Coding for Low-Bitrate Speech Coding](https://mlsanity.com/paper/2211.02506/) (2022)
- [High Quality Audio Coding with MDCTNet](https://mlsanity.com/paper/2212.04583/) (2022)

## Built on by (most cited first, 2 of 2)

- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs](https://mlsanity.com/paper/2602.15491/) (2026)

## Abstract (arXiv)

This paper explores the integration of model-based and data-driven approaches within the realm of neural speech and audio coding systems. It highlights the challenges posed by the subjective evaluation processes of speech and audio codecs and discusses the limitations of purely data-driven approaches, which often require inefficiently large architectures to match the performance of model-based methods. The study presents hybrid systems as a viable solution, offering significant improvements to the performance of conventional codecs through meticulously chosen design enhancements. Specifically, it introduces a neural network-based signal enhancer designed to post-process existing codecs' output, along with the autoencoder-based end-to-end models and LPCNet--hybrid systems that combine linear predictive coding (LPC) with neural networks. Furthermore, the paper delves into predictive models operating within custom feature spaces (TF-Codec) or predefined transform domains (MDCTNet) and examines the use of psychoacoustically calibrated loss functions to train end-to-end neural audio codecs. Through these investigations, the paper demonstrates the potential of hybrid systems to advance the field of speech and audio coding by bridging the gap between traditional model-based approaches and modern data-driven techniques.