# AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec

arXiv: https://arxiv.org/abs/2305.16608 · HTML: https://mlsanity.com/paper/2305.16608/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2023.

## Built on by (most cited first, 30 of 47)

- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [MiMo-Audio: Audio Language Models are Few-Shot Learners](https://mlsanity.com/paper/2512.23808/) (2025)
- [Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations](https://mlsanity.com/paper/2402.12786/) (2024)
- [FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec](https://mlsanity.com/paper/2309.07405/) (2023)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation](https://mlsanity.com/paper/2506.19774/) (2025)
- [Autoregressive Diffusion Transformer for Text-to-Speech Synthesis](https://mlsanity.com/paper/2406.05551/) (2024)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio](https://mlsanity.com/paper/2405.04880/) (2024)
- [RepCodec: A Speech Representation Codec for Speech Tokenization](https://mlsanity.com/paper/2309.00169/) (2023)
- [AV-RIR: Audio-Visual Room Impulse Response Estimation](https://mlsanity.com/paper/2312.00834/) (2023)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [FlowDec: A flow-based full-band general audio codec with high perceptual quality](https://mlsanity.com/paper/2503.01485/) (2025)
- [Audio Super-Resolution with Latent Bridge Models](https://mlsanity.com/paper/2509.17609/) (2025)
- [MOSS-TTS Technical Report](https://mlsanity.com/paper/2603.18090/) (2026)
- [SSDM: Scalable Speech Dysfluency Modeling](https://mlsanity.com/paper/2408.16221/) (2024)
- [DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models](https://mlsanity.com/paper/2410.24177/) (2024)
- [FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation](https://mlsanity.com/paper/2509.16195/) (2025)
- [Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations](https://mlsanity.com/paper/2407.03495/) (2024)
- [Apollo: Band-sequence Modeling for High-Quality Audio Restoration](https://mlsanity.com/paper/2409.08514/) (2024)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)
- [Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio](https://mlsanity.com/paper/2406.08112/) (2024)
- [PAST: Phonetic-Acoustic Speech Tokenizer](https://mlsanity.com/paper/2505.14470/) (2025)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [Conan: A Chunkwise Online Network for Zero-Shot Adaptive Voice Conversion](https://mlsanity.com/paper/2507.14534/) (2025)
- [PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders](https://mlsanity.com/paper/2404.02702/) (2024)
- [Continuous Speech Tokenizer in Text To Speech](https://mlsanity.com/paper/2410.17081/) (2024)
- [PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models](https://mlsanity.com/paper/2406.07801/) (2024)

## Abstract (arXiv)

A good audio codec for live applications such as telecommunication is characterized by three key properties: (1) compression, i.e. the bitrate that is required to transmit the signal should be as low as possible; (2) latency, i.e. encoding and decoding the signal needs to be fast enough to enable communication without or with only minimal noticeable delay; and (3) reconstruction quality of the signal. In this work, we propose an open-source, streamable, and real-time neural audio codec that achieves strong performance along all three axes: it can reconstruct highly natural sounding 48~kHz speech signals while operating at only 12~kbps and running with less than 6~ms (GPU)/10~ms (CPU) latency. An efficient training paradigm is also demonstrated for developing such neural audio codecs for real-world scenarios. Both objective and subjective evaluations using the VCTK corpus are provided. To sum up, AudioDec is a well-developed plug-and-play benchmark for audio codec applications.