# ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgation

arXiv: https://arxiv.org/abs/2410.18932 · HTML: https://mlsanity.com/paper/2410.18932/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Visual Acoustic Matching](https://mlsanity.com/paper/2202.06875/) (2022)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)
- [Audio Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2303.07522/) (2023)
- [Learning Audio-Visual Dereverberation](https://mlsanity.com/paper/2106.07732/) (2021)
- [Learning State-Aware Visual Representations from Audible Interactions](https://mlsanity.com/paper/2209.13583/) (2022)
- [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022)
- [Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark](https://mlsanity.com/paper/2403.18821/) (2024)
- [Swoosh! Rattle! Thump! -- Actions that Sound](https://mlsanity.com/paper/2007.01851/) (2020)
- [That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation](https://mlsanity.com/paper/2210.01116/) (2022)
- [SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos](https://mlsanity.com/paper/2404.05206/) (2024)
- [Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction](https://mlsanity.com/paper/2405.02821/) (2024)
- [Learning Visual-Audio Representations for Voice-Controlled Robots](https://mlsanity.com/paper/2109.02823/) (2021)
- [Blind as a bat: audible echolocation on small robots](https://mlsanity.com/paper/2301.08327/) (2023)

## Abstract (arXiv)

We propose Audio Noise Awareness using Visuals of Indoors for NAVIgation for quieter robot path planning. While humans are naturally aware of the noise they make and its impact on those around them, robots currently lack this awareness. A key challenge in achieving audio awareness for robots is estimating how loud will the robot's actions be at a listener's location? Since sound depends upon the geometry and material composition of rooms, we train the robot to passively perceive loudness using visual observations of indoor environments. To this end, we generate data on how loud an 'impulse' sounds at different listener locations in simulated homes, and train our Acoustic Noise Predictor (ANP). Next, we collect acoustic profiles corresponding to different actions for navigation. Unifying ANP with action acoustics, we demonstrate experiments with wheeled (Hello Robot Stretch) and legged (Unitree Go2) robots so that these robots adhere to the noise constraints of the environment. See code and data at https://anavi-corl24.github.io/