# SNAC: Multi-Scale Neural Audio Codec

arXiv: https://arxiv.org/abs/2410.14411 · HTML: https://mlsanity.com/paper/2410.14411/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation](https://mlsanity.com/paper/2106.07889/) (2021)
- [The challenge of realistic music generation: modelling raw audio at scale](https://mlsanity.com/paper/1806.10474/) (2018)
- [Low Bit-Rate Speech Coding with VQ-VAE and a WaveNet Decoder](https://mlsanity.com/paper/1910.06464/) (2019)
- [End-to-End Optimized Speech Coding with Deep Neural Networks](https://mlsanity.com/paper/1710.09064/) (2017)

## Built on by (most cited first, 30 of 43)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Step-Audio 2 Technical Report](https://mlsanity.com/paper/2507.16632/) (2025)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model](https://mlsanity.com/paper/2505.03739/) (2025)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models](https://mlsanity.com/paper/2508.15827/) (2025)
- [Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition](https://mlsanity.com/paper/2412.09501/) (2024)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling](https://mlsanity.com/paper/2504.07053/) (2025)
- [WavReward: Spoken Dialogue Models With Generalist Reward Evaluators](https://mlsanity.com/paper/2505.09558/) (2025)
- [MagiCodec: Simple Masked Gaussian-Injected Codec for High-Fidelity Reconstruction and Generation](https://mlsanity.com/paper/2506.00385/) (2025)
- [SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec](https://mlsanity.com/paper/2508.02849/) (2025)
- [Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate](https://mlsanity.com/paper/2505.16845/) (2025)
- [Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy](https://mlsanity.com/paper/2506.22023/) (2025)
- [SonoEdit: Null-Space Constrained Knowledge Editing for Pronunciation Correction in LLM-Based TTS](https://mlsanity.com/paper/2601.17086/) (2026)
- [M*: A Modular, Extensible, Serving System for Multimodal Models](https://mlsanity.com/paper/2606.12688/) (2026)
- [Spectrogram Patch Codec: A 2D Block-Quantized VQ-VAE and HiFi-GAN for Neural Speech Coding](https://mlsanity.com/paper/2509.02244/) (2025)
- [UniSRCodec: Unified and Low-Bitrate Single Codebook Codec with Sub-Band Reconstruction](https://mlsanity.com/paper/2601.02776/) (2026)
- [MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Model](https://mlsanity.com/paper/2501.05787/) (2025)
- [StarVC: A Unified Auto-Regressive Framework for Joint Text and Speech Generation in Voice Conversion](https://mlsanity.com/paper/2506.02414/) (2025)
- [DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE](https://mlsanity.com/paper/2506.21864/) (2025)
- [Latent-Mark: An Audio Watermark Robust to Neural Codec Compression](https://mlsanity.com/paper/2603.05310/) (2026)
- [VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing](https://mlsanity.com/paper/2605.06765/) (2026)
- [NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching](https://mlsanity.com/paper/2610.00397/) (2026)
- [As Good as It KAN Get: High-Fidelity Audio Representation](https://mlsanity.com/paper/2503.02585/) (2025)
- [Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions](https://mlsanity.com/paper/2509.24457/) (2025)
- [Learning Relationships Between Separate Audio Tracks for Creative Applications](https://mlsanity.com/paper/2509.25296/) (2025)
- [Adapting Neural Audio Codecs to EEG](https://mlsanity.com/paper/2511.23142/) (2025)
- [Improving X-Codec-2.0 for Multi-Lingual Speech: 25 Hz Latent Rate and 24 kHz Sampling](https://mlsanity.com/paper/2601.20185/) (2026)

## Abstract (arXiv)

Neural audio codecs have recently gained popularity because they can represent audio signals with high fidelity at very low bitrates, making it feasible to use language modeling approaches for audio generation and understanding. Residual Vector Quantization (RVQ) has become the standard technique for neural audio compression using a cascade of VQ codebooks. This paper proposes the Multi-Scale Neural Audio Codec, a simple extension of RVQ where the quantizers can operate at different temporal resolutions. By applying a hierarchy of quantizers at variable frame rates, the codec adapts to the audio structure across multiple timescales. This leads to more efficient compression, as demonstrated by extensive objective and subjective evaluations. The code and model weights are open-sourced at https://github.com/hubertsiuzdak/snac.