# Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark

arXiv: https://arxiv.org/abs/2403.18821 · HTML: https://mlsanity.com/paper/2403.18821/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [Atlas: End-to-End 3D Scene Reconstruction from Posed Images](https://mlsanity.com/paper/2003.10432/) (2020)
- [Pyroomacoustics: A Python package for audio room simulations and array processing algorithms](https://mlsanity.com/paper/1710.04196/) (2017)
- [Neural 3D Scene Reconstruction with the Manhattan-world Assumption](https://mlsanity.com/paper/2205.02836/) (2022)
- [Habitat-Matterport 3D Semantics Dataset](https://mlsanity.com/paper/2210.05633/) (2022)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [VR-NeRF: High-Fidelity Virtualized Walkable Spaces](https://mlsanity.com/paper/2311.02542/) (2023)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Visual Acoustic Matching](https://mlsanity.com/paper/2202.06875/) (2022)
- [FAST-RIR: Fast neural diffuse room impulse response generator](https://mlsanity.com/paper/2110.04057/) (2021)
- [Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis](https://mlsanity.com/paper/2103.14201/) (2021)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)
- [Learning Audio-Visual Dereverberation](https://mlsanity.com/paper/2106.07732/) (2021)
- [A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality](https://mlsanity.com/paper/1810.08860/) (2018)
- [SING: Symbol-to-Instrument Neural Generator](https://mlsanity.com/paper/1810.09785/) (2018)
- [MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenes](https://mlsanity.com/paper/2205.09248/) (2022)
- [Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields](https://mlsanity.com/paper/2309.15977/) (2023)
- [AV-RIR: Audio-Visual Room Impulse Response Estimation](https://mlsanity.com/paper/2312.00834/) (2023)

## Built on by (most cited first, 20 of 20)

- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [Images that Sound: Composing Images and Sounds on a Single Canvas](https://mlsanity.com/paper/2405.12221/) (2024)
- [AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis](https://mlsanity.com/paper/2406.08920/) (2024)
- [NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fields](https://mlsanity.com/paper/2405.18213/) (2024)
- [Hearing Anywhere in Any Environment](https://mlsanity.com/paper/2504.10746/) (2025)
- [Multimodal Spatial Language Maps for Robot Navigation and Manipulation](https://mlsanity.com/paper/2506.06862/) (2025)
- [Differentiable Room Acoustic Rendering with Multi-View Vision Priors](https://mlsanity.com/paper/2504.21847/) (2025)
- [SOAF: Scene Occlusion-aware Neural Acoustic Field](https://mlsanity.com/paper/2407.02264/) (2024)
- [SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding](https://mlsanity.com/paper/2504.05576/) (2025)
- [Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness](https://mlsanity.com/paper/2504.16936/) (2025)
- [Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes](https://mlsanity.com/paper/2506.09989/) (2025)
- [Few-shot Acoustic Synthesis with Multimodal Flow Matching](https://mlsanity.com/paper/2603.19176/) (2026)
- [Direction-Aware Neural Acoustic Fields for Few-Shot Interpolation of Ambisonic Impulse Responses](https://mlsanity.com/paper/2505.13617/) (2025)
- [Resounding Acoustic Fields with Reciprocity](https://mlsanity.com/paper/2510.20602/) (2025)
- [VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learning](https://mlsanity.com/paper/2511.20422/) (2025)
- [ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgation](https://mlsanity.com/paper/2410.18932/) (2024)
- [Data Augmentation Using Neural Acoustic Fields With Retrieval-Augmented Pre-training](https://mlsanity.com/paper/2504.14409/) (2025)
- [AV-Unified: A Unified Framework for Audio-visual Scene Understanding](https://mlsanity.com/paper/2603.06530/) (2026)
- [EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction](https://mlsanity.com/paper/2605.28101/) (2026)
- [Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds](https://mlsanity.com/paper/2608.00463/) (2026)

## Abstract (arXiv)

We present a new dataset called Real Acoustic Fields (RAF) that captures real acoustic room data from multiple modalities. The dataset includes high-quality and densely captured room impulse response data paired with multi-view images, and precise 6DoF pose tracking data for sound emitters and listeners in the rooms. We used this dataset to evaluate existing methods for novel-view acoustic synthesis and impulse response generation which previously relied on synthetic data. In our evaluation, we thoroughly assessed existing audio and audio-visual models against multiple criteria and proposed settings to enhance their performance on real-world data. We also conducted experiments to investigate the impact of incorporating visual data (i.e., images and depth) into neural acoustic field models. Additionally, we demonstrated the effectiveness of a simple sim2real approach, where a model is pre-trained with simulated data and fine-tuned with sparse real-world data, resulting in significant improvements in the few-shot learning approach. RAF is the first dataset to provide densely captured room acoustic data, making it an ideal resource for researchers working on audio and audio-visual neural acoustic field modeling techniques. Demos and datasets are available on our project page: https://facebookresearch.github.io/real-acoustic-fields/