# NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference

arXiv: https://arxiv.org/abs/2508.05835 · HTML: https://mlsanity.com/paper/2508.05835/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global context](https://mlsanity.com/paper/2110.04410/) (2021)
- [Hi-Fi Multi-Speaker English TTS Dataset](https://mlsanity.com/paper/2104.01497/) (2021)
- [TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio](https://mlsanity.com/paper/2304.01448/) (2023)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Scaling Transformers for Low-Bitrate High-Quality Speech Coding](https://mlsanity.com/paper/2411.19842/) (2024)
- [Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2009.14153/) (2020)
- [Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance](https://mlsanity.com/paper/2502.05236/) (2025)
- [TS3-Codec: Transformer-Based Simple Streaming Single Codec](https://mlsanity.com/paper/2411.18803/) (2024)
- [ECAPA2: A Hybrid Neural Network Architecture and Training Strategy for Robust Speaker Embeddings](https://mlsanity.com/paper/2401.08342/) (2024)
- [Textless Direct Speech-to-Speech Translation with Discrete Speech Representation](https://mlsanity.com/paper/2211.00115/) (2022)
- [APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding](https://mlsanity.com/paper/2402.10533/) (2024)
- [Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference](https://mlsanity.com/paper/2409.12117/) (2024)
- [Gull: A Generative Multifunctional Audio Codec](https://mlsanity.com/paper/2404.04947/) (2024)
- [An Intra-BRNN and GB-RVQ Based END-TO-END Neural Audio Codec](https://mlsanity.com/paper/2402.01271/) (2024)

## Built on by (most cited first, 15 of 15)

- [Chronological Thinking in Full-Duplex Spoken Dialogue Language Models](https://mlsanity.com/paper/2510.05150/) (2025)
- [FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation](https://mlsanity.com/paper/2509.16195/) (2025)
- [F-Actor: Controllable Conversational Behaviour in Full-Duplex Models](https://mlsanity.com/paper/2601.11329/) (2026)
- [Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec](https://mlsanity.com/paper/2603.05887/) (2026)
- [Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization](https://mlsanity.com/paper/2509.21718/) (2025)
- [MagpieTTS-LF: Inference-Time Long-Form Speech Generation Without Training on Long-Form data](https://mlsanity.com/paper/2606.18485/) (2026)
- [Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generation](https://mlsanity.com/paper/2509.19592/) (2025)
- [IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2606.06559/) (2026)
- [Probing Low Frame Rate Degradation in Neural Audio Codecs](https://mlsanity.com/paper/2606.16969/) (2026)
- [OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL](https://mlsanity.com/paper/2606.30356/) (2026)
- [ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding](https://mlsanity.com/paper/2609.11642/) (2026)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)
- [Controlling Backchannels in Streamable Full-duplex Models](https://mlsanity.com/paper/2609.29418/) (2026)
- [All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2609.30416/) (2026)
- [RUBATO: Decoupled Clocks Allow Full-Duplex Models to Think as Fast as They Can](https://mlsanity.com/paper/2610.14299/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced audio processing by leveraging audio codecs to discretize audio into tokens, enabling the application of language modeling techniques to speech data. However, existing audio codecs often operate at high frame rates, leading to slow training and inference, particularly for autoregressive models. To address this, there is growing interest in low frame-rate audio codecs, which reduce the number of autoregressive steps required to generate one second of audio. In this paper, we conduct ablation studies to examine the impact of frame rate, bitrate, and causality on codec reconstruction quality. Based on our findings, we introduce NanoCodec, a state-of-the-art audio codec that achieves high-quality compression at just 12.5 frames per second (FPS). NanoCodec outperforms related works across various bitrate ranges, establishing a new benchmark for low-latency and efficient Speech LLM training and inference.