# OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

arXiv: https://arxiv.org/abs/2609.23407 · HTML: https://mlsanity.com/paper/2609.23407/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 62 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [Seedance 2.0: Advancing Video Generation for World Complexity](https://mlsanity.com/paper/2604.14148/) (2026)
- [StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling](https://mlsanity.com/paper/2507.05240/) (2025)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Scaling Data Generation in Vision-and-Language Navigation](https://mlsanity.com/paper/2307.15644/) (2023)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Ola: Pushing the Frontiers of Omni-Modal Language Model](https://mlsanity.com/paper/2502.04328/) (2025)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.17221/) (2025)
- [Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation](https://mlsanity.com/paper/2512.08186/) (2025)
- [OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM](https://mlsanity.com/paper/2510.15870/) (2025)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [LangNav: Language as a Perceptual Representation for Navigation](https://mlsanity.com/paper/2310.07889/) (2023)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [BAT: Learning to Reason about Spatial Sounds with Large Language Models](https://mlsanity.com/paper/2402.01591/) (2024)
- [Audio Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2303.07522/) (2023)
- [Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence](https://mlsanity.com/paper/2604.24954/) (2026)
- [AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments](https://mlsanity.com/paper/2210.07940/) (2022)
- [Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation](https://mlsanity.com/paper/2311.13209/) (2023)
- [Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparities](https://mlsanity.com/paper/2507.13019/) (2025)

## Built on by (most cited first, 2 of 2)

- [Qwen3.8-Omni: Towards Native Omni-Modal Agents](https://mlsanity.com/paper/2609.25611/) (2026)
- [EmbodiedCLBench: Evaluating Continual Learning for Self-Evolving Embodied Agents](https://mlsanity.com/paper/2610.13021/) (2026)

## Abstract (arXiv)

Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. Our code and data will be available in https://github.com/PKU-VaLuE-Lab/OmniEcho/tree/main