# Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

arXiv: https://arxiv.org/abs/2609.00727 · HTML: https://mlsanity.com/paper/2609.00727/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation](https://mlsanity.com/paper/2404.14396/) (2024)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Kosmos-G: Generating Images in Context with Multimodal Large Language Models](https://mlsanity.com/paper/2310.02992/) (2023)
- [EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis](https://mlsanity.com/paper/2308.05725/) (2023)
- [Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling](https://mlsanity.com/paper/2301.00591/) (2023)
- [Mellow: a small audio language model for reasoning](https://mlsanity.com/paper/2503.08540/) (2025)
- [Prompting Audios Using Acoustic Properties For Emotion Representation](https://mlsanity.com/paper/2310.02298/) (2023)
- [FlashLabs Chroma 1.0: A Real-Time End-to-End Spoken Dialogue Model with Personalized Voice Cloning](https://mlsanity.com/paper/2601.11141/) (2026)
- [Towards End-to-End Private Automatic Speaker Recognition](https://mlsanity.com/paper/2206.11750/) (2022)

## Built on by (most cited first, 2 of 2)

- [Thinking in Depth, Speaking Directly: Recurrent Latent Reasoning for Paralinguistically Grounded Spoken Dialogue](https://mlsanity.com/paper/2609.37818/) (2026)
- [Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change](https://mlsanity.com/paper/2609.35922/) (2026)

## Abstract (arXiv)

Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.