# Continuous Audio Language Models

arXiv: https://arxiv.org/abs/2509.06926 · HTML: https://mlsanity.com/paper/2509.06926/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2025.

## Built on by (most cited first, 10 of 10)

- [SoundReactor: Frame-level Online Video-to-Audio Generation](https://mlsanity.com/paper/2510.02110/) (2025)
- [Scaling Properties of Continuous Diffusion Spoken Language Models](https://mlsanity.com/paper/2604.24416/) (2026)
- [Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators](https://mlsanity.com/paper/2605.22717/) (2026)
- [Next-Token Prediction Learns Generalisable Representations of Sleep Physiology](https://mlsanity.com/paper/2606.09605/) (2026)
- [Quantifying the Generation Modality Gap in Speech-Text Language Models](https://mlsanity.com/paper/2609.14743/) (2026)
- [Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding](https://mlsanity.com/paper/2606.27320/) (2026)
- [CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents](https://mlsanity.com/paper/2608.08638/) (2026)
- [Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesis](https://mlsanity.com/paper/2609.25411/) (2026)
- [Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing](https://mlsanity.com/paper/2610.00063/) (2026)
- [Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions](https://mlsanity.com/paper/2610.17461/) (2026)

## Abstract (arXiv)

Audio Language Models (ALM) have emerged as the dominant paradigm for speech and music generation by representing audio as sequences of discrete tokens. Yet, unlike text tokens, which are invertible, audio tokens are extracted from lossy codecs with a limited bitrate. As a consequence, increasing audio quality requires generating more tokens, which imposes a trade-off between fidelity and computational cost. We address this issue by studying Continuous Audio Language Models (CALM). These models instantiate a large Transformer backbone that produces a contextual embedding at every timestep. This sequential information then conditions an MLP that generates the next continuous frame of an audio VAE through consistency modeling. By avoiding lossy compression, CALM achieves higher quality at lower computational cost than their discrete counterpart. Experiments on speech and music demonstrate improved efficiency and fidelity over state-of-the-art discrete audio language models, facilitating lightweight, high-quality audio generation. Samples are available at hf.co/spaces/kyutai/calm-samples. Finally, we release Pocket TTS, an open-source 100M-parameter text-to-speech model that can run faster than real time on a laptop CPU: github.com/kyutai-labs/pocket-tts.