# BatVision: Learning to See 3D Spatial Layout with Two Ears

arXiv: https://arxiv.org/abs/1912.07011 · HTML: https://mlsanity.com/paper/1912.07011/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2019.

## Built on by (most cited first, 26 of 26)

- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video](https://mlsanity.com/paper/2111.10882/) (2021)
- [Measuring Acoustics with Collaborative Multiple Agents](https://mlsanity.com/paper/2310.05368/) (2023)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [Structure from Silence: Learning Scene Structure from Ambient Sound](https://mlsanity.com/paper/2111.05846/) (2021)
- [Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention](https://mlsanity.com/paper/2111.08046/) (2021)
- [NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields](https://mlsanity.com/paper/2405.18213/) (2024)
- [Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations](https://mlsanity.com/paper/2301.02184/) (2023)
- [Hearing Anywhere in Any Environment](https://mlsanity.com/paper/2504.10746/) (2025)
- [Estimating Visual Information From Audio Through Manifold Learning](https://mlsanity.com/paper/2208.02337/) (2022)
- [SoundCam: A Dataset for Finding Humans Using Room Acoustics](https://mlsanity.com/paper/2311.03517/) (2023)
- [Differentiable Room Acoustic Rendering with Multi-View Vision Priors](https://mlsanity.com/paper/2504.21847/) (2025)
- [Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds](https://mlsanity.com/paper/2109.02763/) (2021)
- [PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound](https://mlsanity.com/paper/2112.00216/) (2021)
- [Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision](https://mlsanity.com/paper/2207.01136/) (2022)
- [3-Dimensional Sonic Phase-invariant Echo Localization](https://mlsanity.com/paper/2306.08281/) (2023)
- [The Un-Kidnappable Robot: Acoustic Localization of Sneaking People](https://mlsanity.com/paper/2310.03743/) (2023)
- [Few-shot Acoustic Synthesis with Multimodal Flow Matching](https://mlsanity.com/paper/2603.19176/) (2026)
- [EchoPT: A Pretrained Transformer Architecture that Predicts 2D In-Air Sonar Images for Mobile Robotics](https://mlsanity.com/paper/2405.12573/) (2024)
- [BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Sounds](https://mlsanity.com/paper/2503.00389/) (2025)
- [How Robot Dogs See the Unseeable: Improving Visual Interpretability via Peering for Exploratory Robots](https://mlsanity.com/paper/2511.16262/) (2025)
- [StofNet: Super-resolution Time of Flight Network](https://mlsanity.com/paper/2308.12009/) (2023)
- [Estimating Indoor Scene Depth Maps from Ultrasonic Echoes](https://mlsanity.com/paper/2409.03336/) (2024)
- [Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video](https://mlsanity.com/paper/2512.12165/) (2025)

## Abstract (arXiv)

Many species have evolved advanced non-visual perception while artificial systems fall behind. Radar and ultrasound complement camera-based vision but they are often too costly and complex to set up for very limited information gain. In nature, sound is used effectively by bats, dolphins, whales, and humans for navigation and communication. However, it is unclear how to best harness sound for machine perception. Inspired by bats' echolocation mechanism, we design a low-cost BatVision system that is capable of seeing the 3D spatial layout of space ahead by just listening with two ears. Our system emits short chirps from a speaker and records returning echoes through microphones in an artificial human pinnae pair. During training, we additionally use a stereo camera to capture color images for calculating scene depths. We train a model to predict depth maps and even grayscale images from the sound alone. During testing, our trained BatVision provides surprisingly good predictions of 2D visual scenes from two 1D audio signals. Such a sound to vision system would benefit robot navigation and machine vision, especially in low-light or no-light conditions. Our code and data are publicly available.