# Communicating Sound Through Natural Language

arXiv: https://arxiv.org/abs/2605.08750 · HTML: https://mlsanity.com/paper/2605.08750/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 9 May 2026.

## Builds on (most cited first, 18 of 18)

- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation](https://mlsanity.com/paper/2411.18138/) (2024)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling](https://mlsanity.com/paper/2504.07053/) (2025)
- [TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling](https://mlsanity.com/paper/2508.16790/) (2025)
- [Semantic Communications using Foundation Models: Design Approaches and Open Issues](https://mlsanity.com/paper/2309.13315/) (2023)
- [UniAudio 2.0: A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization](https://mlsanity.com/paper/2602.04683/) (2026)
- [Robust Steganography from Large Language Models](https://mlsanity.com/paper/2504.08977/) (2025)
- [SILA: Signal-to-Language Augmentation for Enhanced Control in Text-to-Audio Generation](https://mlsanity.com/paper/2412.09789/) (2024)
- [LLMs can hide text in other text of the same length](https://mlsanity.com/paper/2510.20075/) (2025)
- [Word Embeddings for Automatic Equalization in Audio Mixing](https://mlsanity.com/paper/2202.08898/) (2022)
- [L2· M = C2 Large Language Models are Covert Channels](https://mlsanity.com/paper/2405.15652/) (2024)
- [TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment](https://mlsanity.com/paper/2602.23068/) (2026)

## Abstract (arXiv)

Natural language is widely used to describe, prompt, and control audio systems, but rarely serves as the representation carrying audio itself. We introduce lexical acoustic coding (LAC), a framework in which pre-trained LLM sender and receiver agents transmit sound through natural language. Under fixed system prompts, the agents write their own analysis and synthesis code, communicating only through a lexical sentence, shared vocabulary, and optional symbolic music structure. The sender analyzes an input waveform into interpretable, non-learned acoustic descriptors, quantizes each with a feature-specific interval vocabulary, and verbalizes the lexical code as English. The receiver parses the sentence back into lexical-acoustic constraints and renders a waveform through closed-loop refinement. The transmitted text serves as both a rich caption and as the transport representation itself. We frame LAC as a finite-rate lossy quantizer, exposing trade-offs between vocabulary size, rate, and fidelity. Experiments on short sounds and symbolic music transfer show that plain text preserves measurable acoustic structure while remaining interpretable, editable, and native to LLM-mediated communication.