mlsanityData to 14 Oct 2026

2.5D Visual Sound

arXiv 1812.04204 ↗First posted 11 Dec 2018cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound---the visual stream helps "lift" the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: http://vision.cs.utexas.edu/projects/2.5D_visual_sound/

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 7; 2023: 15; 2024: 13; 2025: 10; 2026: 3.

63 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2021Localizing Visual Sounds the Hard Waycited by 170
  3. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  4. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  5. 2022Localizing Visual Sounds the Easy Waycited by 82
  6. 2021Robust Audio-Visual Instance Discriminationcited by 80
  7. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  8. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  9. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  10. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  11. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  12. 2022Visual Acoustic Matchingcited by 45
38 more
  1. 2021Audio-Visual Synchronisation in the wildcited by 40
  2. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  3. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  4. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  5. 2023Novel-View Acoustic Synthesiscited by 33
  6. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  7. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  8. 2021Learning Audio-Visual Dereverberationcited by 26
  9. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  10. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  11. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  12. 2023A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognitioncited by 21
  13. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  14. 2023DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 20
  15. 2023Weakly-Supervised Audio-Visual Segmentationcited by 19
  16. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  17. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  18. 2022Active Audio-Visual Separation of Dynamic Sound Sourcescited by 16
  19. 2023Self-Supervised Visual Acoustic Matchingcited by 14
  20. 2024SoundingActions: Learning How Actions Sound from Narrated Egocentric Videoscited by 11
  21. 2025ViSAGe: Video-to-Spatial Audio Generationcited by 11
  22. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 9
  23. 2023RealImpact: A Dataset of Impact Sound Fields for Real Objectscited by 9
  24. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  25. 2023LAVSS: Location-Guided Audio-Visual Spatial Audio Separationcited by 8
  26. 2025Hearing Anywhere in Any Environmentcited by 8
  27. 2024Audio-visual Generalized Zero-shot Learning the Easy Waycited by 7
  28. 2024Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Modelscited by 6
  29. 2024Answering Diverse Questions via Text Attached with Key Audio-Visual Cluescited by 5
  30. 2024Semantic Grouping Network for Audio Source Separationcited by 5
  31. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  32. 2025Learning to Highlight Audio by Watching Moviescited by 5
  33. 2021Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Soundscited by 4
  34. 2024Aligning Audio-Visual Joint Representations with an Agentic Workflowcited by 4
  35. 2024Multi-scale Multi-instance Visual Sound Localization and Segmentationcited by 3
  36. 2025SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Bindingcited by 3
  37. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  38. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3

50 most cited of 63 All 63 (JSON)