# SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Binding

arXiv: https://arxiv.org/abs/2504.05576 · HTML: https://mlsanity.com/paper/2504.05576/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [VR-NeRF: High-Fidelity Virtualized Walkable Spaces](https://mlsanity.com/paper/2311.02542/) (2023)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [2.5D Visual Sound](https://mlsanity.com/paper/1812.04204/) (2018)
- [Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation](https://mlsanity.com/paper/2104.02026/) (2021)
- [Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation](https://mlsanity.com/paper/2007.09902/) (2020)
- [Audio-Visual Grouping Network for Sound Localization from Mixtures](https://mlsanity.com/paper/2303.17056/) (2023)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Visually Informed Binaural Audio Generation without Binaural Audios](https://mlsanity.com/paper/2104.06162/) (2021)
- [Visual Acoustic Matching](https://mlsanity.com/paper/2202.06875/) (2022)
- [Vision-Infused Deep Audio Inpainting](https://mlsanity.com/paper/1910.10997/) (2019)
- [FAST-RIR: Fast neural diffuse room impulse response generator](https://mlsanity.com/paper/2110.04057/) (2021)
- [Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis](https://mlsanity.com/paper/2103.14201/) (2021)
- [Novel-View Acoustic Synthesis](https://mlsanity.com/paper/2301.08730/) (2023)

## Built on by (most cited first, 3 of 3)

- [Resounding Acoustic Fields with Reciprocity](https://mlsanity.com/paper/2510.20602/) (2025)
- [Materialistic RIR: Material Conditioned Realistic RIR Generation](https://mlsanity.com/paper/2604.21119/) (2026)
- [Audible World Models: Spatially Aware Sound Generation for 3D Worlds](https://mlsanity.com/paper/2609.38444/) (2026)

## Abstract (arXiv)

We introduce SoundVista, a method to generate the ambient sound of an arbitrary scene at novel viewpoints. Given a pre-acquired recording of the scene from sparsely distributed microphones, SoundVista can synthesize the sound of that scene from an unseen target viewpoint. The method learns the underlying acoustic transfer function that relates the signals acquired at the distributed microphones to the signal at the target viewpoint, using a limited number of known recordings. Unlike existing works, our method does not require constraints or prior knowledge of sound source details. Moreover, our method efficiently adapts to diverse room layouts, reference microphone configurations and unseen environments. To enable this, we introduce a visual-acoustic binding module that learns visual embeddings linked with local acoustic properties from panoramic RGB and depth data. We first leverage these embeddings to optimize the placement of reference microphones in any given scene. During synthesis, we leverage multiple embeddings extracted from reference locations to get adaptive weights for their contribution, conditioned on target viewpoint. We benchmark the task on both publicly available data and real-world settings. We demonstrate significant improvements over existing methods.