mlsanityData to 7 Oct 2026

Self-Supervised Generation of Spatial Audio for 360 Video

arXiv 1809.02587 ↗First posted 7 Sep 2018cs.SD cs.CV cs.LGcited by 93

Not on the map. Why?

Abstract (arXiv)

We introduce an approach to convert mono audio recorded by a 360 video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360 video viewing, but spatial audio microphones are still rare in current 360 video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis of audio and 360 video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360 videos uploaded with spatial audio. During training, ground-truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360 video and a mono audio track.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 13; 2023: 21; 2024: 17; 2025: 13; 2026: 7.

93 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  3. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  4. 2022Localizing Visual Sounds the Easy Waycited by 81
  5. 2021Robust Audio-Visual Instance Discriminationcited by 80
  6. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  7. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  8. 2023A Comprehensive Survey on Segment Anything Model for Vision and Beyondcited by 64
  9. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  10. 2023AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentationcited by 52
  11. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  12. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
38 more
  1. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  2. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  3. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
  4. 2022Deep Learning for Omnidirectional Vision: A Survey and New Perspectivescited by 38
  5. 2024Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentationcited by 38
  6. 2023Novel-View Acoustic Synthesiscited by 33
  7. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  8. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  9. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  10. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  11. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  12. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  13. 2021Learning Audio-Visual Dereverberationcited by 26
  14. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  15. 2021Beyond Image to Depth: Improving Depth Prediction using Echoescited by 24
  16. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  17. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  18. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  19. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  20. 2023A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognitioncited by 21
  21. 2023Measuring Acoustics with Collaborative Multiple Agentscited by 21
  22. 2023Audio-Visual Instance Segmentationcited by 21
  23. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  24. 2023DiffAVA: Personalized Text-to-Audio Generation with Visual Alignmentcited by 19
  25. 2023Weakly-Supervised Audio-Visual Segmentationcited by 19
  26. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  27. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  28. 2021Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attentioncited by 17
  29. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  30. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  31. 2022MarginNCE: Robust Sound Localization with a Negative Margincited by 16
  32. 2021Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generationcited by 14
  33. 2024AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesiscited by 14
  34. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  35. 2022Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separationcited by 11
  36. 2025ViSAGe: Video-to-Spatial Audio Generationcited by 11
  37. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  38. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 8

50 most cited of 93 All 93 (JSON)