mlsanityData to 14 Oct 2026

SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Sound

arXiv 2406.06612 ↗First posted 6 Jun 2024cs.CV cs.LG cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

Generating combined visual and auditory sensory experiences is critical for the consumption of immersive content. Recent advances in neural generative models have enabled the creation of high-resolution content across multiple modalities such as images, text, speech, and videos. Despite these successes, there remains a significant gap in the generation of high-quality spatial audio that complements generated visual content. Furthermore, current audio generation models excel in either generating natural audio or speech or music but fall short in integrating spatial audio cues necessary for immersive experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that decomposes the task into (1) identifying visual regions of interest; (2) locating these elements in 3D space; (3) generating mono-audio for each; and (4) integrating them into spatial audio. Using our framework, we demonstrate compelling results for generating spatial audio for high-quality videos, images, and dynamic images from the internet, as well as media generated by learned approaches.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 75 of its 104 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

75 of 104 references.

References · 75

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  8. 2023Segment Anythingcited by 8,234
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Generative Adversarial Networkscited by 5,627
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  6. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  11. 2023Simple and Controllable Music Generationcited by 488
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  13. 2017Look, Listen and Learncited by 458
  14. 2023MusicLM: Generating Music From Textcited by 448
  15. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  16. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  17. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  18. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  19. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  20. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  21. 2022AudioGen: Textually Guided Audio Generationcited by 295
  22. 2018The Sound of Pixelscited by 259
  23. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  24. 2017Objects that Soundcited by 256
  25. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  26. 2016Lip Reading Sentences in the Wildcited by 228
  27. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  28. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  29. 2023Text-To-4D Dynamic Scene Generationcited by 205
  30. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  31. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  32. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  33. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  34. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  35. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  36. 2021Taming Visually Guided Sound Generationcited by 136
  37. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  38. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 132

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 6.

10 papers.

  1. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  2. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  3. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  4. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3
  5. 2025In-the-wild Audio Spatialization with Flexible Text-guided Localizationcited by 1
  6. 2026FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clipscited by 1
  7. 2026SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generationcited by 0
  8. 2026Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worldscited by 0
  9. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worldscited by 0
  10. 2026Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationcited by 0