mlsanityData to 7 Oct 2026

Novel-View Acoustic Synthesis

arXiv 2301.08730 ↗First posted 20 Jan 2023cs.CV cs.SD eess.AScited by 33

Not on the map. Why?

Abstract (arXiv)

We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint? We propose a neural rendering approach: Visually-Guided Acoustic Synthesis (ViGAS) network that learns to synthesize the sound of an arbitrary point in space by analyzing the input audio-visual cues. To benchmark this task, we collect two first-of-their-kind large-scale multi-view audio-visual datasets, one synthetic and one real. We show that our model successfully reasons about the spatial cues and synthesizes faithful audio on both datasets. To our knowledge, this work represents the very first formulation, dataset, and approach to solve the novel-view acoustic synthesis task, which has exciting potential applications ranging from AR/VR to art and design. Unlocked by this work, we believe that the future of novel-view synthesis is in multi-modal learning from videos.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 20 Jan 2023.

Builds on

37 of 63 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  4. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  7. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  8. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  9. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  10. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  11. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
25 more
  1. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  4. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  5. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  6. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  7. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  8. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  9. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  10. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  11. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  12. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  13. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 81
  14. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  15. 2022ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformerscited by 70
  16. 2022Learning Neural Acoustic Fieldscited by 65
  17. 20182.5D Visual Soundcited by 63
  18. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  19. 2022Visual Acoustic Matchingcited by 45
  20. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  21. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  22. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  23. 2021Learning Audio-Visual Dereverberationcited by 26
  24. 2022MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenescited by 23
  25. 2022Deep Impulse Responses: Estimating and Parameterizing Filters with Deep Networkscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 12; 2025: 8; 2026: 5.

33 papers.

Sort:

By area:

  1. 2023NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functionscited by 65
  2. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  3. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  4. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
  5. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  6. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  7. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  8. 2023Self-Supervised Visual Acoustic Matchingcited by 14
  9. 2024AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesiscited by 14
  10. 2024Hearing Anything Anywherecited by 12
  11. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  12. 2025Hearing Anywhere in Any Environmentcited by 8
21 more
  1. 2023Radar Fields: An Extension of Radiance Fields to SARcited by 7
  2. 2023Novel-View Acoustic Synthesis from 3D Reconstructed Roomscited by 6
  3. 2024EgoSonics: Generating Synchronized Audio for Silent Egocentric Videoscited by 6
  4. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  5. 2025BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Modelscited by 5
  6. 2024Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speechcited by 4
  7. 2025How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenescited by 4
  8. 2024SOAF: Scene Occlusion-aware Neural Acoustic Fieldcited by 3
  9. 2025SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Bindingcited by 3
  10. 2025Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenescited by 2
  11. 2024TeX-NeRF: Neural Radiance Fields from Pseudo-TeX Visioncited by 1
  12. 2024MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modelingcited by 0
  13. 2024ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgationcited by 0
  14. 2024Zero-Shot Mono-to-Binaural Speech Synthesiscited by 0
  15. 2025AV-Surf: Surface-Enhanced Geometry-Aware Novel-View Acoustic Synthesiscited by 0
  16. 2025Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Videocited by 0
  17. 2026Materialistic RIR: Material Conditioned Realistic RIR Generationcited by 0
  18. 20263DAE: Binaural Quality Assessment for Audio Novel View Synthesis with Spatial Maps and Benchmarkcited by 0
  19. 2026Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worldscited by 0
  20. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worldscited by 0
  21. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0