mlsanityData to 14 Oct 2026

Structure from Silence: Learning Scene Structure from Ambient Sound

arXiv 2111.05846 ↗First posted 10 Nov 2021cs.SD cs.CV cs.MMcited by 17

Not on the map. Why?

Abstract (arXiv)

From whirling ceiling fans to ticking clocks, the sounds that we hear subtly vary as we move through a scene. We ask whether these ambient sounds convey information about 3D scene structure and, if so, whether they provide a useful learning signal for multimodal models. To study this, we collect a dataset of paired audio and RGB-D recordings from a variety of quiet indoor scenes. We then train models that estimate the distance to nearby walls, given only audio as input. We also use these recordings to learn multimodal representations through self-supervision, by training a network to associate images with their corresponding sounds. These results suggest that ambient sound conveys a surprising amount of information about scene structure, and that it is a useful signal for learning multimodal features.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 51 of its 86 references that are in the map. First posted on arXiv 10 Nov 2021.

Builds on

51 of 86 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  8. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  9. 2017Look, Listen and Learncited by 458
  10. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  11. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  12. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
38 more
  1. 2018The Sound of Pixelscited by 259
  2. 2017Objects that Soundcited by 256
  3. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  5. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
  6. 2016Learning to Navigate in Complex Environmentscited by 198
  7. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  8. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  9. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  10. 2016Single-Image Depth Perception in the Wildcited by 180
  11. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  12. 2021Localizing Visual Sounds the Hard Waycited by 170
  13. 2019The Sound of Motionscited by 146
  14. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 146
  15. 2015Visually Indicated Soundscited by 128
  16. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  17. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  18. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  19. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  20. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  21. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 96
  22. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  23. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 83
  24. 2020Semantic Audio-Visual Navigationcited by 82
  25. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  26. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  27. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  28. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  29. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 55
  30. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  31. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  32. 2021Extreme Rotation Estimation using Dense Correlation Volumescited by 42
  33. 2017Learning to Fly by Crashingcited by 41
  34. 2020Audio-Visual Floorplan Reconstructioncited by 32
  35. 2019BatVision: Learning to See 3D Spatial Layout with Two Earscited by 26
  36. 2018Learning Single-Image Depth from Videos using Quality Assessment Networkscited by 25
  37. 2020Associative3D: Volumetric Reconstruction from Sparse Viewscited by 18
  38. 2017Reflection-Aware Sound Source Localizationcited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 4; 2024: 3; 2025: 4; 2026: 1.

17 papers.

Sort:

By area:

  1. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  2. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  3. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  4. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  5. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  6. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  7. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  8. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  9. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 14
  10. 2022That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulationcited by 12
  11. 2025Hearing Anywhere in Any Environmentcited by 8
  12. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
5 more
  1. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  2. 2024Improving Generalization of Speech Separation in Real-World Scenarios: Strategies in Simulation, Optimization, and Evaluationcited by 3
  3. 2023The Un-Kidnappable Robot: Acoustic Localization of Sneaking Peoplecited by 2
  4. 2025SpeechOp: Inference-Time Task Composition for Generative Speech Processingcited by 0
  5. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0