mlsanityData to 14 Oct 2026

ViSAGe: Video-to-Spatial Audio Generation

arXiv 2506.12199 ↗First posted 13 Jun 2025cs.SD cs.AI eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

Spatial audio is essential for enhancing the immersiveness of audio-visual experiences, yet its production typically demands complex recording systems and specialized expertise. In this work, we address a novel problem of generating first-order ambisonics, a widely used spatial audio format, directly from silent videos. To support this task, we introduce YT-Ambigen, a dataset comprising 102K 5-second YouTube video clips paired with corresponding first-order ambisonics. We also propose new evaluation metrics to assess the spatial aspect of generated audio based on audio energy maps and saliency metrics. Furthermore, we present Video-to-Spatial Audio Generation (ViSAGe), an end-to-end framework that generates first-order ambisonics from silent video frames by leveraging CLIP visual features, autoregressive neural audio codec modeling with both directional and visual guidance. Experimental results demonstrate that ViSAGe produces plausible and coherent first-order ambisonics, outperforming two-stage approaches consisting of video-to-audio generation and audio spatialization. Qualitative examples further illustrate that ViSAGe generates temporally aligned high-quality spatial audio that adapts to viewpoint changes.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 51 of its 68 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

51 of 68 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2022Video Diffusion Modelscited by 2,305
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  12. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
38 more
  1. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  5. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  6. 2022High Fidelity Neural Audio Compressioncited by 687
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  9. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  10. 2023CoTracker: It is Better to Track Togethercited by 522
  11. 2023Simple and Controllable Music Generationcited by 488
  12. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  13. 2022AudioGen: Textually Guided Audio Generationcited by 295
  14. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  15. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  16. 2021Taming Visually Guided Sound Generationcited by 136
  17. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  18. 2015Visually Indicated Soundscited by 128
  19. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  20. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  21. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  22. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  23. 2020Generating Visually Aligned Sound from Videoscited by 67
  24. 2018Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videoscited by 66
  25. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  26. 20182.5D Visual Soundcited by 63
  27. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  28. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  29. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  30. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 41
  31. 2023FoleyGen: Visually-Guided Audio Generationcited by 40
  32. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  33. 2020Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Soundscited by 26
  34. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 26
  35. 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
  36. 2022Panoramic Vision Transformer for Saliency Detection in 360deg Videoscited by 15
  37. 2024DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidancecited by 11
  38. 2024ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Modelcited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2025MOSPA: Human Motion Generation Driven by Spatial Audiocited by 12
  2. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  3. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3
  4. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  5. 2026Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformercited by 1
  6. 2026DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videoscited by 0
  7. 2026DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videoscited by 0
  8. 2026Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worldscited by 0
  9. 2026PhysWave: Physics-Guided Latent Diffusion Models for Controllable Spatial Audio Generationcited by 0
  10. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worldscited by 0
  11. 2026Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generationcited by 0