# Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance

arXiv: https://arxiv.org/abs/2510.10444 · HTML: https://mlsanity.com/paper/2510.10444/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 12 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.04779/) (2025)
- [Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks](https://mlsanity.com/paper/2411.05361/) (2024)
- [MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models](https://mlsanity.com/paper/2408.01337/) (2024)
- [AHELM: A Holistic Evaluation of Audio-Language Models](https://mlsanity.com/paper/2508.21376/) (2025)

## Built on by (most cited first, 24 of 24)

- [Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox](https://mlsanity.com/paper/2605.27772/) (2026)
- [When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration](https://mlsanity.com/paper/2602.11488/) (2026)
- [The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASRrightarrowLLM Pipelines?](https://mlsanity.com/paper/2602.17598/) (2026)
- [Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering](https://mlsanity.com/paper/2603.06854/) (2026)
- [Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization](https://mlsanity.com/paper/2604.06138/) (2026)
- [Real-Time Voice AI Hears but Does Not Listen](https://mlsanity.com/paper/2606.26083/) (2026)
- [Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation](https://mlsanity.com/paper/2607.13477/) (2026)
- [Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does](https://mlsanity.com/paper/2608.19515/) (2026)
- [Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs](https://mlsanity.com/paper/2602.23136/) (2026)
- [DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models](https://mlsanity.com/paper/2603.18048/) (2026)
- [SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models](https://mlsanity.com/paper/2606.25990/) (2026)
- [Separating Decision-Rule Misalignment from Readout-Coverage Limitations in Speech Language Models](https://mlsanity.com/paper/2608.06409/) (2026)
- [Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation](https://mlsanity.com/paper/2608.06718/) (2026)
- [CLASH: Counterfactual Auditing of Lexical and Prosodic Reliance in Spoken Sarcasm Detection](https://mlsanity.com/paper/2609.16582/) (2026)
- [Enhancing Audio Reasoning via Semantic Summary Prediction](https://mlsanity.com/paper/2609.20849/) (2026)
- [MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages](https://mlsanity.com/paper/2511.04914/) (2025)
- [Interleaved Speech Language Models Latently Work In Text](https://mlsanity.com/paper/2606.22473/) (2026)
- [Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models](https://mlsanity.com/paper/2607.11801/) (2026)
- [Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models](https://mlsanity.com/paper/2608.17102/) (2026)
- [Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs](https://mlsanity.com/paper/2609.05871/) (2026)
- [Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study](https://mlsanity.com/paper/2609.26823/) (2026)
- [Almost Human, Except When It Matters: VoxParity and the Decisions a Voice Should Change](https://mlsanity.com/paper/2609.35922/) (2026)
- [EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidance](https://mlsanity.com/paper/2609.38182/) (2026)
- [DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs](https://mlsanity.com/paper/2610.08085/) (2026)

## Abstract (arXiv)

Understanding emotion from speech requires sensitivity to both lexical and acoustic cues. However, it remains unclear whether large audio language models (LALMs) genuinely process acoustic information or rely primarily on lexical content. We present LISTEN (Lexical vs. Acoustic Speech Test for Emotion in Narratives), a controlled benchmark designed to disentangle lexical reliance from acoustic sensitivity in emotion understanding. Across evaluations of six state-of-the-art LALMs, we observe a consistent lexical dominance. Models predict "neutral" when lexical cues are neutral or absent, show limited gains under cue alignment, and fail to classify distinct emotions under cue conflict. In paralinguistic settings, performance approaches chance. These results indicate that current LALMs largely "transcribe" rather than "listen," relying heavily on lexical semantics while underutilizing acoustic cues. LISTEN offers a principled framework for assessing emotion understanding in multimodal models.