mlsanityData to 7 Oct 2026

Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds

arXiv 2608.00463 ↗First posted 1 Aug 2026cs.CV cs.MM cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

3D Gaussian Splatting (3DGS) turns captured or generated imagery into photorealistic 3D world simulations that users can freely explore, yet these worlds remain silent. Because existing audio generation methods condition on a single image or viewpoint, their sound is tied to that observation and cannot stay consistent while a listener moves. We introduce the task of generating a spatially consistent soundscape for a given 3DGS world through auditory grounding, identifying which objects in the world should emit sound and anchoring each to a persistent 3D position, and present Scene2Sound, a training-free framework built on this grounding. From the input world alone, our pipeline selects viewpoints that jointly cover the scene, identifies sound-emitting objects with a vision-language model, and associates the multi-view detections into 3D instances through Gaussian set matching, which measures the overlap between the Gaussian sets that render each detection. Each source then receives generated audio that a standard object-based audio engine spatializes in real time at arbitrary listener poses. We further propose two spatial-consistency metrics, one testing whether rendered audio responds consistently to listener motion and one testing whether the claimed sources are supported by views held out from their placement. On a curated set of generated 3DGS worlds and on 3DGS scenes generated from real-world 360-degree captures, Scene2Sound preserves the audio quality of strong per-viewpoint baselines while remaining spatially consistent where per-viewpoint and single-panorama pipelines do not, and a user study confirms the perceptual benefit. Project page: https://masaki-lmd.github.io/scene2sound/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

46 of 58 references.

References · 46

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  5. 2025SAM 3: Segment Anything with Conceptscited by 835
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  8. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  9. 2019Clotho: An Audio Captioning Datasetcited by 354
  10. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  11. 2024gsplat: An Open-Source Library for Gaussian Splattingcited by 279
  12. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
34 more
  1. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  2. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  3. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  4. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  5. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
  6. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  7. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  8. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  9. 2022Learning Neural Acoustic Fieldscited by 65
  10. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  11. 2023Novel-View Acoustic Synthesiscited by 33
  12. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  13. 2024Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splattingcited by 30
  14. 2024GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splattingcited by 25
  15. 2025Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentationcited by 23
  16. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
  17. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  18. 2024FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generationcited by 19
  19. 2025OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splattingcited by 19
  20. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  21. 2024AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesiscited by 14
  22. 2024Acoustic Volume Rendering for Neural Impulse Response Fieldscited by 14
  23. 2024Hearing Anything Anywherecited by 12
  24. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11
  25. 2025ViSAGe: Video-to-Spatial Audio Generationcited by 11
  26. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  27. 2025Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphscited by 10
  28. 2025PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generationcited by 7
  29. 2025AG2aussian: Anchor-Graph Structured Gaussian Splatting for Instance-Level 3D Scene Understanding and Editingcited by 4
  30. 2024DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splattingcited by 3
  31. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  32. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3
  33. 2025Sound Scene Synthesis at the DCASE 2024 Challengecited by 2
  34. 2025Visual Acoustic Fieldscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worlds
  2. 2026WorldSonus: Bringing Sound to Worlds
  3. 2026DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos
  4. 2026DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos
  5. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
  6. 2026HelixWorld: A Real-time Interactive Audio-Visual World Model
  7. 2025AudioScene: Integrating Object-Event Audio into 3D Scenes
  8. 2026Physics-Aware Novel-View Acoustic Synthesis with Vision-Language Priors and 3D Acoustic Environment Modeling