# Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigation

arXiv: https://arxiv.org/abs/2308.10306 · HTML: https://mlsanity.com/paper/2308.10306/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 30 of 43)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Anomaly Detection via Reverse Distillation from One-Class Embedding](https://mlsanity.com/paper/2201.10703/) (2022)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [Audio Adversarial Examples: Targeted Attacks on Speech-to-Text](https://mlsanity.com/paper/1801.01944/) (2018)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Cross-modal Map Learning for Vision and Language Navigation](https://mlsanity.com/paper/2203.05137/) (2022)
- [Distilling Policy Distillation](https://mlsanity.com/paper/1902.02186/) (2019)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.11591/) (2022)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022)
- [Active Visual Information Gathering for Vision-Language Navigation](https://mlsanity.com/paper/2007.08037/) (2020)
- [Auxiliary Tasks Speed Up Learning PointGoal Navigation](https://mlsanity.com/paper/2007.04561/) (2020)

## Built on by (most cited first, 19 of 19)

- [Volumetric Environment Representation for Vision-Language Navigation](https://mlsanity.com/paper/2403.14158/) (2024)
- [Navigation Instruction Generation with BEV Perception and Large Language Models](https://mlsanity.com/paper/2407.15087/) (2024)
- [Vision-Language Navigation with Energy-Based Policy](https://mlsanity.com/paper/2410.14250/) (2024)
- [Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation](https://mlsanity.com/paper/2509.16924/) (2025)
- [Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments](https://mlsanity.com/paper/2509.22698/) (2025)
- [Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks](https://mlsanity.com/paper/2509.25652/) (2025)
- [Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigation](https://mlsanity.com/paper/2509.21377/) (2025)
- [Embodied Navigation with Auxiliary Task of Action Description Prediction](https://mlsanity.com/paper/2510.21809/) (2025)
- [OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents](https://mlsanity.com/paper/2609.23407/) (2026)
- [Audio-Guided Visual Perception for Audio-Visual Navigation](https://mlsanity.com/paper/2510.11760/) (2025)
- [Residual Cross-Modal Fusion Networks for Audio-Visual Navigation](https://mlsanity.com/paper/2601.08868/) (2026)
- [Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion](https://mlsanity.com/paper/2407.10373/) (2024)
- [SkeNa: Learning to Navigate Unseen Environments Based on Abstract Hand-Drawn Maps](https://mlsanity.com/paper/2508.03053/) (2025)
- [Audio Spatially-Guided Fusion for Audio-Visual Navigation](https://mlsanity.com/paper/2604.02389/) (2026)
- [Spatial-Aware Conditioned Fusion for Audio-Visual Navigation](https://mlsanity.com/paper/2604.02390/) (2026)
- [Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction](https://mlsanity.com/paper/2604.05007/) (2026)
- [CTAN: Cycle-Temporal Attention Network for Embodied Audio-Visual Navigation](https://mlsanity.com/paper/2609.17420/) (2026)
- [Transformer-Based Token Fusion and Dynamic Graph Planning for Audio-Visual Navigation](https://mlsanity.com/paper/2609.17421/) (2026)
- [Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation](https://mlsanity.com/paper/2610.14077/) (2026)

## Abstract (arXiv)

Audio-visual navigation is an audio-targeted wayfinding task where a robot agent is entailed to travel a never-before-seen 3D environment towards the sounding source. In this article, we present ORAN, an omnidirectional audio-visual navigator based on cross-task navigation skill transfer. In particular, ORAN sharpens its two basic abilities for a such challenging task, namely wayfinding and audio-visual information gathering. First, ORAN is trained with a confidence-aware cross-task policy distillation (CCPD) strategy. CCPD transfers the fundamental, point-to-point wayfinding skill that is well trained on the large-scale PointGoal task to ORAN, so as to help ORAN to better master audio-visual navigation with far fewer training samples. To improve the efficiency of knowledge transfer and address the domain gap, CCPD is made to be adaptive to the decision confidence of the teacher policy. Second, ORAN is equipped with an omnidirectional information gathering (OIG) mechanism, i.e., gleaning visual-acoustic observations from different directions before decision-making. As a result, ORAN yields more robust navigation behaviour. Taking CCPD and OIG together, ORAN significantly outperforms previous competitors. After the model ensemble, we got 1st in Soundspaces Challenge 2022, improving SPL and SR by 53% and 35% relatively.