SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
On the map: LLM reasoning → Audio-language model reasoning
Abstract (arXiv)
3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLMs) and benchmarks, which predominantly focus on static or 2D scenes. We introduce SAVVY-Bench, the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of relationships involving static and moving objects, and requires fine-grained temporal grounding, consistent 3D localization, and multi-modal annotation. To tackle this challenge, we propose SAVVY, a novel training-free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs as well as other audio-visual methods to track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi-modal queried object trajectories and converts them into a unified global dynamic map. Using the constructed map, a final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state-of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.
Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 76 references that are in the map. First posted on arXiv 4 Jun 2025.
Builds on
62 of 76 references.
Closest ancestors · 2
Same direction, newest first
- 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models4 Mar
- 2024BAT: Learning to Reason about Spatial Sounds with Large Language Models2 Feb
Other references · 60
- 2023Visual Instruction Tuningcited by 7,792
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
- 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
38 more
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2023VideoChat: Chat-Centric Video Understandingcited by 851
- 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
- 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
- 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
- 2021Image Segmentation Using Text and Image Promptscited by 509
- 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
- 2022Visual Language Maps for Robot Navigationcited by 362
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
- 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
- 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
- 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
- 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
- 2025Kimi-Audio Technical Reportcited by 210
- 2023Listen, Think, and Understandcited by 206
- 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
- 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
- 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
- 2023Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Camerascited by 139
- 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
- 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
- 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
- 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
- 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
- 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
- 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
- 2018Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networkscited by 84
- 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
- 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
- 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
- 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
- 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2025: 6; 2026: 8.
14 papers.
Sort:
By area:
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
- 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 17
- 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
- 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 8
- 2026Do Joint Audio-Video Generation Models Understand Physics?cited by 8
- 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
- 2025DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Modelscited by 4
- 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
- 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedycited by 1
- 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agentscited by 1
- 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Modelscited by 1
- 2026Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottleneckscited by 0
2 more
- 2026OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents26 Sep 2026
- 2026SceneBind: Binding What and Where Across Vision, Audio and Language16 Jul 2026
- 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents28 May 2026
- 2026Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks18 May 2026
- 2026Do Joint Audio-Video Generation Models Understand Physics?8 May 2026
- 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models17 Apr 2026
- 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery7 Apr 2026
- 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videos15 Feb 2026
- 2025A Unified Definition of Hallucination: It's The World Model, Stupid!25 Dec 2025
- 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space17 Dec 2025
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks29 Oct 2025
- 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy22 Sep 2025
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
- 2025DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Modelscited by 4
- 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
- 2026OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agentscited by 0
- 2026OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents26 Sep 2026
- 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space17 Dec 2025
- 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks29 Oct 2025
- 2025DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models17 Sep 2025
- 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 17
- 2026Do Joint Audio-Video Generation Models Understand Physics?cited by 8
- 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedycited by 1
- 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agentscited by 1
- 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Modelscited by 1
- 2026SceneBind: Binding What and Where Across Vision, Audio and Languagecited by 0
- 2026SceneBind: Binding What and Where Across Vision, Audio and Language16 Jul 2026
- 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents28 May 2026
- 2026Do Joint Audio-Video Generation Models Understand Physics?8 May 2026
- 2026MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models17 Apr 2026
- 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery7 Apr 2026
- 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy22 Sep 2025
Similar, not cited
Close in wording, no citation either way
- 2026SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding
- 2026The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
- 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence
- 2025Spatial Audio Motion Understanding and Reasoning
- 2025AudioScene: Integrating Object-Event Audio into 3D Scenes
- 2026Spatial Audio Question Answering and Reasoning on Dynamic Source Movements
- 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
- 2026JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments