# Can Large Language Models Understand Spatial Audio?

arXiv: https://arxiv.org/abs/2406.07914 · HTML: https://mlsanity.com/paper/2406.07914/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2024.

## Built on by (most cited first, 15 of 15)

- [video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2506.15220/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions](https://mlsanity.com/paper/2503.20290/) (2025)
- [Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization](https://mlsanity.com/paper/2410.06682/) (2024)
- [SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models](https://mlsanity.com/paper/2511.06606/) (2025)
- [Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding](https://mlsanity.com/paper/2606.10738/) (2026)
- [PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs](https://mlsanity.com/paper/2601.21124/) (2026)
- [DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models](https://mlsanity.com/paper/2509.13927/) (2025)
- [Sci-Phi: A Large Language Model Spatial Audio Descriptor](https://mlsanity.com/paper/2510.05542/) (2025)
- [Spatial Audio Processing with Large Language Model on Wearable Devices](https://mlsanity.com/paper/2504.08907/) (2025)
- [Teaching Physical Awareness to LLMs through Sounds](https://mlsanity.com/paper/2506.08524/) (2025)
- [Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition](https://mlsanity.com/paper/2506.14973/) (2025)
- [Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources](https://mlsanity.com/paper/2606.14141/) (2026)
- [SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning](https://mlsanity.com/paper/2610.17131/) (2026)

## Abstract (arXiv)

This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs' advanced cognitive and inferential abilities, the aim is to enhance understanding of 3D environments via audio. We study 3 spatial audio tasks: sound source localization (SSL), far-field speech recognition (FSR), and localisation-informed speech extraction (LSE), achieving notable progress in each task. For SSL, our approach achieves an MAE of 2.70circ on the Spatial LibriSpeech dataset, substantially surpassing the prior benchmark of about 6.60circ. Moreover, our model can employ spatial cues to improve FSR accuracy and execute LSE by selectively attending to sounds originating from a specified direction via text prompts, even amidst overlapping speech. These findings highlight the potential of adapting LLMs to grasp physical audio concepts, paving the way for LLM-based agents in 3D environments.