# BAT: Learning to Reason about Spatial Sounds with Large Language Models

arXiv: https://arxiv.org/abs/2402.01591 · HTML: https://mlsanity.com/paper/2402.01591/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 30 of 37)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT](https://mlsanity.com/paper/2310.04673/) (2023)
- [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024)
- [Telling Left from Right: Learning Spatial Correspondence of Sight and Sound](https://mlsanity.com/paper/2006.06175/) (2020)

## Built on by (most cited first, 30 of 36)

- [video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2506.15220/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Learning Spatially-Aware Language and Audio Embeddings](https://mlsanity.com/paper/2409.11369/) (2024)
- [MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences](https://mlsanity.com/paper/2410.02381/) (2024)
- [SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing](https://mlsanity.com/paper/2506.05414/) (2025)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [MOSPA: Human Motion Generation Driven by Spatial Audio](https://mlsanity.com/paper/2507.11949/) (2025)
- [EgoSound: Benchmarking Sound Understanding in Egocentric Videos](https://mlsanity.com/paper/2602.14122/) (2026)
- [OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis](https://mlsanity.com/paper/2501.04561/) (2025)
- [OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models](https://mlsanity.com/paper/2509.26140/) (2025)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization](https://mlsanity.com/paper/2410.06682/) (2024)
- [SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models](https://mlsanity.com/paper/2511.06606/) (2025)
- [Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding](https://mlsanity.com/paper/2606.10738/) (2026)
- [The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models](https://mlsanity.com/paper/2601.02954/) (2026)
- [PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs](https://mlsanity.com/paper/2601.21124/) (2026)
- [DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models](https://mlsanity.com/paper/2509.13927/) (2025)
- [Sci-Phi: A Large Language Model Spatial Audio Descriptor](https://mlsanity.com/paper/2510.05542/) (2025)
- [Spatial Audio Processing with Large Language Model on Wearable Devices](https://mlsanity.com/paper/2504.08907/) (2025)
- [Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience](https://mlsanity.com/paper/2505.05515/) (2025)
- [Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition](https://mlsanity.com/paper/2506.14973/) (2025)
- [Spatial Audio Question Answering and Reasoning on Dynamic Source Movements](https://mlsanity.com/paper/2602.16334/) (2026)
- [Spatial Audio Motion Understanding and Reasoning](https://mlsanity.com/paper/2509.14666/) (2025)
- [AudioMotionBench: Evaluating Auditory Motion Perception in Audio LLMs](https://mlsanity.com/paper/2511.13273/) (2025)
- [OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents](https://mlsanity.com/paper/2609.23407/) (2026)
- [Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification](https://mlsanity.com/paper/2608.02397/) (2026)
- [Spectro-Temporal Interference Confounds Phase Encoding in Spatial Audio Foundation Models](https://mlsanity.com/paper/2606.14820/) (2026)
- [SceneBind: Binding What and Where Across Vision, Audio and Language](https://mlsanity.com/paper/2607.15265/) (2026)
- [Bearings: Self-Supervised Soundfield Embeddings from First-Order Ambisonics](https://mlsanity.com/paper/2609.23152/) (2026)

## Abstract (arXiv)

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.