# Analysing the Language of Neural Audio Codecs

arXiv: https://arxiv.org/abs/2509.01390 · HTML: https://mlsanity.com/paper/2509.01390/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Builds on (most cited first, 28 of 28)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec](https://mlsanity.com/paper/2305.02765/) (2023)
- [SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models](https://mlsanity.com/paper/2308.16692/) (2023)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling](https://mlsanity.com/paper/2301.00591/) (2023)
- [AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec](https://mlsanity.com/paper/2305.16608/) (2023)
- [FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec](https://mlsanity.com/paper/2309.07405/) (2023)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Towards audio language modeling -- an overview](https://mlsanity.com/paper/2402.13236/) (2024)
- [Codec-SUPERB: An In-Depth Analysis of Sound Codec Models](https://mlsanity.com/paper/2402.13071/) (2024)
- [Analyzing The Language of Visual Tokens](https://mlsanity.com/paper/2411.05001/) (2024)
- [Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations](https://mlsanity.com/paper/2407.03495/) (2024)
- [Towards Audio Codec-based Speech Separation](https://mlsanity.com/paper/2406.12434/) (2024)
- [Do learned speech symbols follow Zipf's law?](https://mlsanity.com/paper/2309.09690/) (2023)

## Built on by (most cited first, 2 of 2)

- [Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks](https://mlsanity.com/paper/2601.12205/) (2026)
- [Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions](https://mlsanity.com/paper/2608.31037/) (2026)

## Abstract (arXiv)

This study presents a comparative analysis of the statistical and linguistic properties of neural audio codecs (NACs). We investigate discrete speech tokens produced by various NAC models, examining their adherence to linguistic statistical laws such as Zipf's law and Heaps' law, as well as their entropy and redundancy. To assess how these token-level properties relate to semantic and acoustic preservation in synthesized speech, we evaluate intelligibility using error rates of automatic speech recognition, and quality using the UTMOS score. Our results reveal that NAC tokens, particularly 3-grams, exhibit language-like statistical patterns. Moreover, these properties, together with measures of information content, are found to correlate with improved performances in speech recognition and resynthesis tasks. These findings offer insights into the structure of NAC token sequences and inform the design of more effective generative speech models.