# VisualEchoes: Spatial Image Representation Learning through Echolocation

arXiv: https://arxiv.org/abs/2005.01616 · HTML: https://mlsanity.com/paper/2005.01616/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 30 of 55)

- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Toward Practical Monocular Indoor Depth Estimation](https://mlsanity.com/paper/2112.02306/) (2021)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [Learning Audio-Visual Dereverberation](https://mlsanity.com/paper/2106.07732/) (2021)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [Can audio-visual integration strengthen robustness under multimodal attacks?](https://mlsanity.com/paper/2104.02000/) (2021)
- [Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video](https://mlsanity.com/paper/2111.10882/) (2021)
- [Measuring Acoustics with Collaborative Multiple Agents](https://mlsanity.com/paper/2310.05368/) (2023)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [Structure from Silence: Learning Scene Structure from Ambient Sound](https://mlsanity.com/paper/2111.05846/) (2021)
- [Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention](https://mlsanity.com/paper/2111.08046/) (2021)
- [Images that Sound: Composing Images and Sounds on a Single Canvas](https://mlsanity.com/paper/2405.12221/) (2024)
- [AdVerb: Visually Guided Audio Dereverberation](https://mlsanity.com/paper/2308.12370/) (2023)
- [Finding Fallen Objects Via Asynchronous Audio-Visual Integration](https://mlsanity.com/paper/2207.03483/) (2022)
- [The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective](https://mlsanity.com/paper/2312.12870/) (2023)
- [RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2309.00655/) (2023)
- [SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos](https://mlsanity.com/paper/2404.05206/) (2024)
- [Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear](https://mlsanity.com/paper/2306.00923/) (2023)
- [RealImpact: A Dataset of Impact Sound Fields for Real Objects](https://mlsanity.com/paper/2306.09944/) (2023)
- [NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields](https://mlsanity.com/paper/2405.18213/) (2024)
- [Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations](https://mlsanity.com/paper/2301.02184/) (2023)
- [Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos](https://mlsanity.com/paper/2307.04760/) (2023)
- [Hearing Anywhere in Any Environment](https://mlsanity.com/paper/2504.10746/) (2025)

## Abstract (arXiv)

Several animal species (e.g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world. We explore the spatial cues contained in echoes and how they can benefit vision tasks that require spatial reasoning. First we capture echo responses in photo-realistic 3D indoor scene environments. Then we propose a novel interaction-based representation learning framework that learns useful visual features via echolocation. We show that the learned image features are useful for multiple downstream vision tasks requiring spatial reasoning---monocular depth estimation, surface normal estimation, and visual navigation---with results comparable or even better than heavily supervised pre-training. Our work opens a new path for representation learning for embodied agents, where supervision comes from interacting with the physical world.