# Audio-Visual Floorplan Reconstruction

arXiv: https://arxiv.org/abs/2012.15470 · HTML: https://mlsanity.com/paper/2012.15470/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 30 of 32)

- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models](https://mlsanity.com/paper/2308.07997/) (2023)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Toward Practical Monocular Indoor Depth Estimation](https://mlsanity.com/paper/2112.02306/) (2021)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [Context Understanding in Computer Vision: A Survey](https://mlsanity.com/paper/2302.05011/) (2023)
- [Measuring Acoustics with Collaborative Multiple Agents](https://mlsanity.com/paper/2310.05368/) (2023)
- [Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2502.12425/) (2025)
- [Pay Self-Attention to Audio-Visual Navigation](https://mlsanity.com/paper/2210.01353/) (2022)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [Structure from Silence: Learning Scene Structure from Ambient Sound](https://mlsanity.com/paper/2111.05846/) (2021)
- [Finding Fallen Objects Via Asynchronous Audio-Visual Integration](https://mlsanity.com/paper/2207.03483/) (2022)
- [Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2310.19559/) (2023)
- [CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models](https://mlsanity.com/paper/2402.13607/) (2024)
- [Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear](https://mlsanity.com/paper/2306.00923/) (2023)
- [RealImpact: A Dataset of Impact Sound Fields for Real Objects](https://mlsanity.com/paper/2306.09944/) (2023)
- [NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields](https://mlsanity.com/paper/2405.18213/) (2024)
- [Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations](https://mlsanity.com/paper/2301.02184/) (2023)
- [Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos](https://mlsanity.com/paper/2307.04760/) (2023)
- [EgoSonics: Generating Synchronized Audio for Silent Egocentric Videos](https://mlsanity.com/paper/2407.20592/) (2024)
- [SoundCam: A Dataset for Finding Humans Using Room Acoustics](https://mlsanity.com/paper/2311.03517/) (2023)
- [UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model](https://mlsanity.com/paper/2511.18845/) (2025)
- [ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling](https://mlsanity.com/paper/2404.16216/) (2024)
- [SALVe: Semantic Alignment Verification for Floorplan Reconstruction from Sparse Panoramas](https://mlsanity.com/paper/2406.19390/) (2024)
- [Few-shot Acoustic Synthesis with Multimodal Flow Matching](https://mlsanity.com/paper/2603.19176/) (2026)
- [PrivDiffuser: Privacy-Guided Diffusion Model for Data Obfuscation in Sensor Networks](https://mlsanity.com/paper/2412.14499/) (2024)
- [Predicting Topological Maps for Visual Navigation in Unexplored Environments](https://mlsanity.com/paper/2211.12649/) (2022)
- [RGI-Net: 3D Room Geometry Inference from Room Impulse Responses With Hidden First-Order Reflections](https://mlsanity.com/paper/2309.01513/) (2023)

## Abstract (arXiv)

Given only a few glimpses of an environment, how much can we infer about its entire floorplan? Existing methods can map only what is visible or immediately apparent from context, and thus require substantial movements through a space to fully map it. We explore how both audio and visual sensing together can provide rapid floorplan reconstruction from limited viewpoints. Audio not only helps sense geometry outside the camera's field of view, but it also reveals the existence of distant freespace (e.g., a dog barking in another room) and suggests the presence of rooms not visible to the camera (e.g., a dishwasher humming in what must be the kitchen to the left). We introduce AV-Map, a novel multi-modal encoder-decoder framework that reasons jointly about audio and vision to reconstruct a floorplan from a short input video sequence. We train our model to predict both the interior structure of the environment and the associated rooms' semantic labels. Our results on 85 large real-world environments show the impact: with just a few glimpses spanning 26% of an area, we can estimate the whole area with 66% accuracy -- substantially better than the state of the art approach for extrapolating visual maps.