# Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation

arXiv: https://arxiv.org/abs/2509.16924 · HTML: https://mlsanity.com/paper/2509.16924/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 21 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [Sound Adversarial Audio-Visual Navigation](https://mlsanity.com/paper/2202.10910/) (2022)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [See, Hear, Explore: Curiosity via Audio-Visual Association](https://mlsanity.com/paper/2007.03669/) (2020)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [Measuring Acoustics with Collaborative Multiple Agents](https://mlsanity.com/paper/2310.05368/) (2023)
- [Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigation](https://mlsanity.com/paper/2308.10306/) (2023)
- [Zero-shot Object Navigation with Vision-Language Models Reasoning](https://mlsanity.com/paper/2410.18570/) (2024)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challenges](https://mlsanity.com/paper/2311.18044/) (2023)
- [DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusion](https://mlsanity.com/paper/2504.21366/) (2025)
- [Learning Semantic-Agnostic and Spatial-Aware Representation for Generalizable Visual-Audio Navigation](https://mlsanity.com/paper/2304.10773/) (2023)
- [Generating Diverse Audio-Visual 360 Soundscapes for Sound Event Localization and Detection](https://mlsanity.com/paper/2504.02988/) (2025)
- [Exploring the Reliability of Foundation Model-Based Frontier Selection in Zero-Shot Object Goal Navigation](https://mlsanity.com/paper/2410.21037/) (2024)

## Built on by (most cited first, 15 of 15)

- [Audio-Visual Intelligence in Large Foundation Models](https://mlsanity.com/paper/2605.04045/) (2026)
- [Audio Spatially-Guided Fusion for Audio-Visual Navigation](https://mlsanity.com/paper/2604.02389/) (2026)
- [Spatial-Aware Conditioned Fusion for Audio-Visual Navigation](https://mlsanity.com/paper/2604.02390/) (2026)
- [Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation](https://mlsanity.com/paper/2604.02391/) (2026)
- [Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction](https://mlsanity.com/paper/2604.05007/) (2026)
- [EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence](https://mlsanity.com/paper/2604.23325/) (2026)
- [ML-SAN: Multi-Level Speaker-Adaptive Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2604.25383/) (2026)
- [MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2607.10599/) (2026)
- [HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models](https://mlsanity.com/paper/2607.13072/) (2026)
- [Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2608.03264/) (2026)
- [CTAN: Cycle-Temporal Attention Network for Embodied Audio-Visual Navigation](https://mlsanity.com/paper/2609.17420/) (2026)
- [Transformer-Based Token Fusion and Dynamic Graph Planning for Audio-Visual Navigation](https://mlsanity.com/paper/2609.17421/) (2026)
- [Talking Head Synthesis with Facial Landmark Guidance via 3D Gaussian Splatting](https://mlsanity.com/paper/2609.17422/) (2026)
- [WS-NeRF: A Mamba-Driven World-State-Aware Adaptive Deblurring Neural Radiance Field](https://mlsanity.com/paper/2609.21391/) (2026)
- [Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation](https://mlsanity.com/paper/2610.14077/) (2026)

## Abstract (arXiv)

In audio-visual navigation (AVN) tasks, an embodied agent must autonomously localize a sound source in unknown and complex 3D environments based on audio-visual signals. Existing methods often rely on static modality fusion strategies and neglect the spatial cues embedded in stereo audio, leading to performance degradation in cluttered or occluded scenes. To address these issues, we propose an end-to-end reinforcement learning-based AVN framework with two key innovations: (1) a Stereo-Aware Attention Module (SAM), which learns and exploits the spatial disparity between left and right audio channels to enhance directional sound perception; and (2) an Audio-Guided Dynamic Fusion Module (AGDF), which dynamically adjusts the fusion ratio between visual and auditory features based on audio cues, thereby improving robustness to environmental changes. Extensive experiments are conducted on two realistic 3D scene datasets, Replica and Matterport3D, demonstrating that our method significantly outperforms existing approaches in terms of navigation success rate and path efficiency. Notably, our model achieves over 40% improvement under audio-only conditions compared to the best-performing baselines. These results highlight the importance of explicitly modeling spatial cues from stereo channels and performing deep multi-modal fusion for robust and efficient audio-visual navigation.