mlsanityData to 14 Oct 2026

SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

arXiv 2206.08312 ↗First posted 16 Jun 2022cs.SD cs.CV cs.MMcited by 95

Not on the map. Why?

Abstract (arXiv)

We introduce SoundSpaces 2.0, a platform for on-the-fly geometry-based audio rendering for 3D environments. Given a 3D mesh of a real-world environment, SoundSpaces can generate highly realistic acoustics for arbitrary sounds captured from arbitrary microphone locations. Together with existing 3D visual assets, it supports an array of audio-visual research tasks, such as audio-visual navigation, mapping, source localization and separation, and acoustic matching. Compared to existing resources, SoundSpaces 2.0 has the advantages of allowing continuous spatial sampling, generalization to novel environments, and configurable microphone and material properties. To our knowledge, this is the first geometry-based acoustic simulation that offers high fidelity and realism while also being fast enough to use for embodied learning. We showcase the simulator's properties and benchmark its performance against real-world audio measurements. In addition, we demonstrate two downstream tasks -- embodied navigation and far-field automatic speech recognition -- and highlight sim2real performance for the latter. SoundSpaces 2.0 is publicly available to facilitate wider research for perceptual systems that can both see and hear.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Builds on 49 of its 99 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

49 of 99 references.

References · 49

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  9. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  10. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  11. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  12. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
37 more
  1. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  2. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  3. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  4. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  5. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  6. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  7. 2019The Sound of Motionscited by 146
  8. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 132
  9. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  10. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  11. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  12. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 96
  13. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  14. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  15. 2020Semantic Audio-Visual Navigationcited by 82
  16. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  17. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  18. 2022Learning Neural Acoustic Fieldscited by 65
  19. 2020A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directionscited by 56
  20. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 55
  21. 2022Sound Adversarial Audio-Visual Navigationcited by 48
  22. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  23. 2021A Survey of Sound Source Localization with Deep Learning Methodscited by 46
  24. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  25. 2022Visual Acoustic Matchingcited by 45
  26. 2018Building and Evaluation of a Real Room Impulse Response Datasetcited by 43
  27. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  28. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  29. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  30. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  31. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  32. 2020Audio-Visual Floorplan Reconstructioncited by 32
  33. 2021Learning Audio-Visual Dereverberationcited by 26
  34. 2022GWA: A Large High-Quality Acoustic Dataset for Audio Processingcited by 25
  35. 2020The Cone of Silence: Speech Separation by Localizationcited by 21
  36. 2019Scene-Aware Audio Rendering via Deep Acoustic Analysiscited by 11
  37. 2020Sound Synthesis, Propagation, and Rendering: A Surveycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 19; 2024: 26; 2025: 23; 2026: 24.

95 papers.

Sort:

By area:

  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  2. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 69
  3. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  4. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  5. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 48
  6. 2022Retrospectives on the Embodied AI Workshopcited by 46
  7. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 41
  8. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  9. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 36
  10. 2023Novel-View Acoustic Synthesiscited by 33
  11. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  12. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
38 more
  1. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  2. 2023A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognitioncited by 21
  3. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  4. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
  5. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  6. 2023Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigationcited by 19
  7. 2022Pay Self-Attention to Audio-Visual Navigationcited by 18
  8. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  9. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  10. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  11. 2025Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigationcited by 15
  12. 2023Self-Supervised Visual Acoustic Matchingcited by 14
  13. 2025Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environmentscited by 14
  14. 2025Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Taskscited by 14
  15. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  16. 2025Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigationcited by 13
  17. 2024Hearing Anything Anywherecited by 12
  18. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  19. 2024TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separationcited by 11
  20. 2023Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenescited by 10
  21. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  22. 2022Towards Improved Room Impulse Response Estimation for Speech Recognitioncited by 9
  23. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 9
  24. 2024NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fieldscited by 9
  25. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  26. 2025Hearing Anywhere in Any Environmentcited by 8
  27. 2026Do Joint Audio-Video Generation Models Understand Physics?cited by 8
  28. 2024Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Predictioncited by 7
  29. 2024Audio-visual Generalized Zero-shot Learning the Easy Waycited by 7
  30. 2023Multi-goal Audio-visual Navigation using Sound Direction Mapcited by 6
  31. 2023Novel-View Acoustic Synthesis from 3D Reconstructed Roomscited by 6
  32. 2025Multimodal Spatial Language Maps for Robot Navigation and Manipulationcited by 6
  33. 2025The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audiocited by 6
  34. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  35. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  36. 2025BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Modelscited by 5
  37. 2025DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Modelscited by 5
  38. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 5

50 most cited of 95 All 95 (JSON)