# Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes

arXiv: https://arxiv.org/abs/2302.02809 · HTML: https://mlsanity.com/paper/2302.02809/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 62 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [Understanding Attention and Generalization in Graph Neural Networks](https://mlsanity.com/paper/1905.02850/) (2019)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [FAST-RIR: Fast neural diffuse room impulse response generator](https://mlsanity.com/paper/2110.04057/) (2021)
- [Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis](https://mlsanity.com/paper/2103.14201/) (2021)
- [Scene-Aware Audio for 360textdegree Videos](https://mlsanity.com/paper/1805.04792/) (2018)
- [GWA: A Large High-Quality Acoustic Dataset for Audio Processing](https://mlsanity.com/paper/2204.01787/) (2022)
- [MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes](https://mlsanity.com/paper/2205.09248/) (2022)
- [Filtered Noise Shaping for Time Domain Room Impulse Response Estimation From Reverberant Speech](https://mlsanity.com/paper/2107.07503/) (2021)
- [TS-RIR: Translated synthetic room impulse responses for speech augmentation](https://mlsanity.com/paper/2103.16804/) (2021)
- [Scene-Aware Audio Rendering via Deep Acoustic Analysis](https://mlsanity.com/paper/1911.06245/) (2019)
- [Improving Reverberant Speech Training Using Diffuse Acoustic Simulation](https://mlsanity.com/paper/1907.03988/) (2019)
- [Towards Improved Room Impulse Response Estimation for Speech Recognition](https://mlsanity.com/paper/2211.04473/) (2022)
- [Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation](https://mlsanity.com/paper/2010.04865/) (2020)
- [Dynamic Coherence-Based EM Ray Tracing Simulations in Vehicular Environments](https://mlsanity.com/paper/2112.07115/) (2021)

## Built on by (most cited first, 10 of 10)

- [AV-RIR: Audio-Visual Room Impulse Response Estimation](https://mlsanity.com/paper/2312.00834/) (2023)
- [AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis](https://mlsanity.com/paper/2406.08920/) (2024)
- [Novel-View Acoustic Synthesis from 3D Reconstructed Rooms](https://mlsanity.com/paper/2310.15130/) (2023)
- [How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenes](https://mlsanity.com/paper/2508.02905/) (2025)
- [ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling](https://mlsanity.com/paper/2404.16216/) (2024)
- [AV-Surf: Surface-Enhanced Geometry-Aware Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2503.12806/) (2025)
- [DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos](https://mlsanity.com/paper/2602.06846/) (2026)
- [DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos](https://mlsanity.com/paper/2604.02781/) (2026)
- [Materialistic RIR: Material Conditioned Realistic RIR Generation](https://mlsanity.com/paper/2604.21119/) (2026)
- [Towards Improving Speaker Distance Estimation through Generative Impulse Response Augmentation](https://mlsanity.com/paper/2605.00721/) (2026)

## Abstract (arXiv)

We present an end-to-end binaural audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based binaural sound propagation method to generate acoustic effects for indoor 3D models of real environments. Any clean audio or dry audio can be convolved with the generated acoustic effects to render audio corresponding to the real environment. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate acoustic effects from the scene latent vector. Our network can handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function for the generator network to incorporate spatial audio effects. Given the source and the listener position, our learning-based binaural sound propagation approach can generate an acoustic effect in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU. We have evaluated the accuracy of our approach with binaural acoustic effects generated using an interactive geometric sound propagation algorithm and captured real acoustic effects / real-world recordings. We also performed a perceptual evaluation and observed that the audio rendered by our approach is more plausible than audio rendered using prior learning-based and geometric-based sound propagation algorithms. We quantitatively evaluated the accuracy of our approach using statistical acoustic parameters, and energy decay curves. The demo videos, code and dataset are available online (https://anton-jeran.github.io/Listen2Scene/).