# RAO-Nav: Probing Omni-Language Models for Zero-shot Semantic Audio-Visual Navigation

arXiv: https://arxiv.org/abs/2609.32224 · HTML: https://mlsanity.com/paper/2609.32224/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation](https://mlsanity.com/paper/2301.13166/) (2023)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models](https://mlsanity.com/paper/2308.07997/) (2023)
- [CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2403.04640/) (2024)
- [Bayesian Relational Memory for Semantic Visual Navigation](https://mlsanity.com/paper/1909.04306/) (2019)
- [A Survey of Sound Source Localization with Deep Learning Methods](https://mlsanity.com/paper/2109.03465/) (2021)
- [Baichuan-Omni Technical Report](https://mlsanity.com/paper/2410.08565/) (2024)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [OctoNav: Towards Generalist Embodied Navigation](https://mlsanity.com/paper/2506.09839/) (2025)
- [Direction of arrival estimation for multiple sound sources using convolutional recurrent neural network](https://mlsanity.com/paper/1710.10059/) (2017)
- [AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments](https://mlsanity.com/paper/2210.07940/) (2022)
- [NavBench: Probing Multimodal Large Language Models for Embodied Navigation](https://mlsanity.com/paper/2506.01031/) (2025)

## Abstract (arXiv)

We explore whether Omni-Language Models (OLMs) can be directly applied to zero-shot Semantic Audio-Visual Navigation (SAVN). Recent work demonstrates that even state-of-the-art specialized models still struggle to achieve generalist multimodal navigation, despite extensive task-specific training. In this paper, we introduce RAO-Nav, short for Reasoning All-in-One OLM, a deployment pipeline for zero-shot SAVN. By leveraging the rich implicit audio-visual knowledge encoded in OLMs, the embodied agent is enabled to ``hear'', ``see'', ``reason'', and ``act'' in the environment. To further elicit the built-in thinking ability of OLMs, we propose a test-time Latent Navigation Reasoning (LNR) module that can be seamlessly integrated into the decoding space. LNR encourages the model to retrieve more target-relevant observations and make effective navigation decisions. Through comprehensive experiments, we show that our framework surpasses existing state-of-the-art baselines on public SAVN benchmarks without using any training data. Moreover, we introduce a new Global Navigation Instruction setting to further evaluate the ability of OLMs to serve as embodied navigation agents. Code: https://github.com/rikeilong/OmniAVNav.