# Pay Self-Attention to Audio-Visual Navigation

arXiv: https://arxiv.org/abs/2210.01353 · HTML: https://mlsanity.com/paper/2210.01353/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 30 of 42)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Occupancy Anticipation for Efficient Exploration and Navigation](https://mlsanity.com/paper/2008.09285/) (2020)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [Learning Affordance Landscapes for Interaction Exploration in 3D Environments](https://mlsanity.com/paper/2008.09241/) (2020)
- [Self-supervised Moving Vehicle Tracking with Stereo Sound](https://mlsanity.com/paper/1910.11760/) (2019)
- [VisualEchoes: Spatial Image Representation Learning through Echolocation](https://mlsanity.com/paper/2005.01616/) (2020)
- [Sound Adversarial Audio-Visual Navigation](https://mlsanity.com/paper/2202.10910/) (2022)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Semantic Curiosity for Active Visual Learning](https://mlsanity.com/paper/2006.09367/) (2020)
- [Visual Acoustic Matching](https://mlsanity.com/paper/2202.06875/) (2022)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [See, Hear, Explore: Curiosity via Audio-Visual Association](https://mlsanity.com/paper/2007.03669/) (2020)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [A Behavioral Approach to Visual Navigation with Graph Localization Networks](https://mlsanity.com/paper/1903.00445/) (2019)
- [Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks](https://mlsanity.com/paper/1709.00507/) (2017)

## Built on by (most cited first, 18 of 18)

- [Measuring Acoustics with Collaborative Multiple Agents](https://mlsanity.com/paper/2310.05368/) (2023)
- [Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation](https://mlsanity.com/paper/2509.16924/) (2025)
- [Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments](https://mlsanity.com/paper/2509.22698/) (2025)
- [Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks](https://mlsanity.com/paper/2509.25652/) (2025)
- [Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigation](https://mlsanity.com/paper/2509.21377/) (2025)
- [Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis](https://mlsanity.com/paper/2501.05474/) (2025)
- [Embodied Navigation with Auxiliary Task of Action Description Prediction](https://mlsanity.com/paper/2510.21809/) (2025)
- [Balancing Task-invariant Interaction and Task-specific Adaptation for Unified Image Fusion](https://mlsanity.com/paper/2504.05164/) (2025)
- [EGSTalker: Real-Time Audio-Driven Talking Head Generation with Efficient Gaussian Deformation](https://mlsanity.com/paper/2510.08587/) (2025)
- [Audio-Guided Visual Perception for Audio-Visual Navigation](https://mlsanity.com/paper/2510.11760/) (2025)
- [Residual Cross-Modal Fusion Networks for Audio-Visual Navigation](https://mlsanity.com/paper/2601.08868/) (2026)
- [AMNet: An Acoustic Model Network for Enhanced Mandarin Speech Synthesis](https://mlsanity.com/paper/2504.09225/) (2025)
- [Audio Spatially-Guided Fusion for Audio-Visual Navigation](https://mlsanity.com/paper/2604.02389/) (2026)
- [Spatial-Aware Conditioned Fusion for Audio-Visual Navigation](https://mlsanity.com/paper/2604.02390/) (2026)
- [Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation](https://mlsanity.com/paper/2604.02391/) (2026)
- [Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction](https://mlsanity.com/paper/2604.05007/) (2026)
- [Transformer-Based Token Fusion and Dynamic Graph Planning for Audio-Visual Navigation](https://mlsanity.com/paper/2609.17421/) (2026)
- [Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation](https://mlsanity.com/paper/2610.14077/) (2026)

## Abstract (arXiv)

Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.