# Towards audio language modeling -- an overview

arXiv: https://arxiv.org/abs/2402.13236 · HTML: https://mlsanity.com/paper/2402.13236/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2024.

## Built on by (most cited first, 30 of 40)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Baichuan-Omni Technical Report](https://mlsanity.com/paper/2410.08565/) (2024)
- [MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2502.18924/) (2025)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Autoregressive Diffusion Transformer for Text-to-Speech Synthesis](https://mlsanity.com/paper/2406.05551/) (2024)
- [DASB - Discrete Audio and Speech Benchmark](https://mlsanity.com/paper/2406.14294/) (2024)
- [Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insights](https://mlsanity.com/paper/2502.12521/) (2025)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs](https://mlsanity.com/paper/2506.23325/) (2025)
- [Preference Alignment Improves Language Model-Based TTS](https://mlsanity.com/paper/2409.12403/) (2024)
- [Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference](https://mlsanity.com/paper/2404.14883/) (2024)
- [MOSS-TTS Technical Report](https://mlsanity.com/paper/2603.18090/) (2026)
- [Game-Time: Evaluating Temporal Dynamics in Spoken Language Models](https://mlsanity.com/paper/2509.26388/) (2025)
- [Live Music Models](https://mlsanity.com/paper/2508.04651/) (2025)
- [DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models](https://mlsanity.com/paper/2410.24177/) (2024)
- [GenVC: Self-Supervised Zero-Shot Voice Conversion](https://mlsanity.com/paper/2502.04519/) (2025)
- [From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data](https://mlsanity.com/paper/2505.20166/) (2025)
- [Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders](https://mlsanity.com/paper/2510.23802/) (2025)
- [AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering](https://mlsanity.com/paper/2601.12248/) (2026)
- [Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations](https://mlsanity.com/paper/2509.15655/) (2025)
- [Learning Sparsity for Effective and Efficient Music Performance Question Answering](https://mlsanity.com/paper/2506.01319/) (2025)
- [Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models](https://mlsanity.com/paper/2604.25591/) (2026)
- [Continuous Speech Tokenizer in Text To Speech](https://mlsanity.com/paper/2410.17081/) (2024)
- [A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation](https://mlsanity.com/paper/2410.22448/) (2024)
- [CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese](https://mlsanity.com/paper/2511.04139/) (2025)
- [VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space](https://mlsanity.com/paper/2411.14642/) (2024)
- [Analysing the Language of Neural Audio Codecs](https://mlsanity.com/paper/2509.01390/) (2025)

## Abstract (arXiv)

Neural audio codecs are initially introduced to compress audio data into compact codes to reduce transmission latency. Researchers recently discovered the potential of codecs as suitable tokenizers for converting continuous audio into discrete codes, which can be employed to develop audio language models (LMs). Numerous high-performance neural audio codecs and codec-based LMs have been developed. The paper aims to provide a thorough and systematic overview of the neural audio codec models and codec-based LMs.