# How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenes

arXiv: https://arxiv.org/abs/2508.02905 · HTML: https://mlsanity.com/paper/2508.02905/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 58 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision](https://mlsanity.com/paper/2211.10439/) (2022)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation](https://mlsanity.com/paper/2305.01836/) (2023)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Visual Acoustic Matching](https://mlsanity.com/paper/2202.06875/) (2022)
- [FAST-RIR: Fast neural diffuse room impulse response generator](https://mlsanity.com/paper/2110.04057/) (2021)
- [Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis](https://mlsanity.com/paper/2103.14201/) (2021)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)
- [Voices Obscured in Complex Environmental Settings (VOICES) corpus](https://mlsanity.com/paper/1804.05053/) (2018)
- [Scene-Aware Audio for 360textdegree Videos](https://mlsanity.com/paper/1805.04792/) (2018)
- [GWA: A Large High-Quality Acoustic Dataset for Audio Processing](https://mlsanity.com/paper/2204.01787/) (2022)
- [Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields](https://mlsanity.com/paper/2309.15977/) (2023)
- [AV-RIR: Audio-Visual Room Impulse Response Estimation](https://mlsanity.com/paper/2312.00834/) (2023)
- [AdVerb: Visually Guided Audio Dereverberation](https://mlsanity.com/paper/2308.12370/) (2023)
- [Self-Supervised Visual Acoustic Matching](https://mlsanity.com/paper/2307.15064/) (2023)
- [TS-RIR: Translated synthetic room impulse responses for speech augmentation](https://mlsanity.com/paper/2103.16804/) (2021)
- [Scene-Aware Audio Rendering via Deep Acoustic Analysis](https://mlsanity.com/paper/1911.06245/) (2019)
- [SoundSpaces: Audio-Visual Navigation in 3D Environments](https://mlsanity.com/paper/1912.11474/) (2019)
- [Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes](https://mlsanity.com/paper/2302.02809/) (2023)
- [Towards Improved Room Impulse Response Estimation for Speech Recognition](https://mlsanity.com/paper/2211.04473/) (2022)
- [Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction](https://mlsanity.com/paper/2405.02821/) (2024)

## Built on by (most cited first, 4 of 4)

- [PAVAS: Physics-Aware Video-to-Audio Synthesis](https://mlsanity.com/paper/2512.08282/) (2025)
- [Few-shot Acoustic Synthesis with Multimodal Flow Matching](https://mlsanity.com/paper/2603.19176/) (2026)
- [Materialistic RIR: Material Conditioned Realistic RIR Generation](https://mlsanity.com/paper/2604.21119/) (2026)
- [MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimation](https://mlsanity.com/paper/2605.00431/) (2026)

## Abstract (arXiv)

How would the sound in a studio change with a carpeted floor and acoustic tiles on the walls? We introduce the task of material-controlled acoustic profile generation, where, given an indoor scene with specific audio-visual characteristics, the goal is to generate a target acoustic profile based on a user-defined material configuration at inference time. We address this task with a novel encoder-decoder approach that encodes the scene's key properties from an audio-visual observation and generates the target Room Impulse Response (RIR) conditioned on the material specifications provided by the user. Our model enables the generation of diverse RIRs based on various material configurations defined dynamically at inference time. To support this task, we create a new benchmark, the Acoustic Wonderland Dataset, designed for developing and evaluating material-aware RIR prediction methods under diverse and challenging settings. Our results demonstrate that the proposed model effectively encodes material information and generates high-fidelity RIRs, outperforming several baselines and state-of-the-art methods.