mlsanityData to 7 Oct 2026

SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

arXiv 2506.05414 ↗First posted 4 Jun 2025cs.CV cs.AI cs.LGcited by 14

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLMs) and benchmarks, which predominantly focus on static or 2D scenes. We introduce SAVVY-Bench, the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of relationships involving static and moving objects, and requires fine-grained temporal grounding, consistent 3D localization, and multi-modal annotation. To tackle this challenge, we propose SAVVY, a novel training-free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs as well as other audio-visual methods to track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi-modal queried object trajectories and converts them into a unified global dynamic map. Using the constructed map, a final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state-of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 76 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

62 of 76 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models4 Mar
  2. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Models2 Feb

Other references · 60

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  8. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  11. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  12. 2021Image Segmentation Using Text and Image Promptscited by 509
  13. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  14. 2022Visual Language Maps for Robot Navigationcited by 362
  15. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  16. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  17. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  18. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  19. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  20. 2025Kimi-Audio Technical Reportcited by 210
  21. 2023Listen, Think, and Understandcited by 206
  22. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  23. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  24. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  25. 2023Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular, Stereo, and RGB-D Camerascited by 139
  26. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  27. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  28. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  29. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  30. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  31. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  32. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  33. 2018Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networkscited by 84
  34. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  35. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  36. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  37. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  38. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding
  2. 2026The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
  3. 2025STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence
  4. 2025Spatial Audio Motion Understanding and Reasoning
  5. 2025AudioScene: Integrating Object-Event Audio into 3D Scenes
  6. 2026Spatial Audio Question Answering and Reasoning on Dynamic Source Movements
  7. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
  8. 2026JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments