# Bringing Interpretability to Neural Audio Codecs

arXiv: https://arxiv.org/abs/2506.04492 · HTML: https://mlsanity.com/paper/2506.04492/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2025.

## Built on by (most cited first, 5 of 5)

- [Latent-Mark: An Audio Watermark Robust to Neural Codec Compression](https://mlsanity.com/paper/2603.05310/) (2026)
- [From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation](https://mlsanity.com/paper/2606.13630/) (2026)
- [Using the Mimi codec for metalinguistic representations](https://mlsanity.com/paper/2608.15799/) (2026)
- [Probing neural audio codecs for distinctions among English nuclear tunes](https://mlsanity.com/paper/2603.14035/) (2026)
- [How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality](https://mlsanity.com/paper/2610.00154/) (2026)

## Abstract (arXiv)

The advent of neural audio codecs has increased in popularity due to their potential for efficiently modeling audio with transformers. Such advanced codecs represent audio from a highly continuous waveform to low-sampled discrete units. In contrast to semantic units, acoustic units may lack interpretability because their training objectives primarily focus on reconstruction performance. This paper proposes a two-step approach to explore the encoding of speech information within the codec tokens. The primary goal of the analysis stage is to gain deeper insight into how speech attributes such as content, identity, and pitch are encoded. The synthesis stage then trains an AnCoGen network for post-hoc explanation of codecs to extract speech attributes from the respective tokens directly.