mlsanityData to 7 Oct 2026

Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

arXiv 2303.11329 ↗First posted 20 Mar 2023cs.CV cs.SD eess.AScited by 18

Not on the map. Why?

Abstract (arXiv)

The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources. We learn to solve these tasks solely through self-supervision. A visual model predicts camera rotation from a pair of images, while an audio model predicts the direction of sound sources from binaural sounds. We train these models to generate predictions that agree with one another. At test time, the models can be deployed independently. To obtain a feature representation that is well-suited to solving this challenging problem, we also propose a method for learning an audio-visual representation through cross-view binauralization: estimating binaural sound from one view, given images and sound from another. Our model can successfully estimate accurate rotations on both real and synthetic scenes, and localize sound sources with accuracy competitive with state-of-the-art self-supervised approaches. Project site: https://ificl.github.io/SLfM/

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 78 of its 111 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

78 of 111 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  8. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  9. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  11. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  12. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
38 more
  1. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  2. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  3. 2022AudioGen: Textually Guided Audio Generationcited by 292
  4. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  5. 2016FMA: A Dataset For Music Analysiscited by 205
  6. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  7. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  8. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  9. 2021Localizing Visual Sounds the Hard Waycited by 169
  10. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  11. 2021Taming Visually Guided Sound Generationcited by 136
  12. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  13. 2015Visually Indicated Soundscited by 128
  14. 2022NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priorscited by 125
  15. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  16. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  17. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 99
  18. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  19. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  20. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  21. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  22. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  23. 2022Localizing Visual Sounds the Easy Waycited by 81
  24. 2020Semantic Audio-Visual Navigationcited by 79
  25. 2022Perspective Fields for Single Image Camera Calibrationcited by 75
  26. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  27. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  28. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  29. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  30. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  31. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  32. 2017Relative Camera Pose Estimation Using Convolutional Neural Networkscited by 47
  33. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  34. 2022Visual Acoustic Matchingcited by 45
  35. 2021UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Renderingcited by 44
  36. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  37. 2021Extreme Rotation Estimation using Dense Correlation Volumescited by 42
  38. 2021Planar Surface Reconstruction from Sparse Viewscited by 40

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 6; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  2. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  3. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  4. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  5. 2024NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fieldscited by 8
  6. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  7. 2023Learning to Taste: A Multimodal Wine Datasetcited by 6
  8. 2023Novel-View Acoustic Synthesis from 3D Reconstructed Roomscited by 6
  9. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  10. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  11. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  12. 2025MultiModal Action Conditioned Video Generationcited by 3
6 more
  1. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  2. 2025Visual Surface Wave Elastography: Revealing Subsurface Physical Properties via Visible Surface Wavescited by 1
  3. 2025Resounding Acoustic Fields with Reciprocitycited by 1
  4. 2024AVS-Net: Audio-Visual Scale Net for Self-supervised Monocular Metric Depth Estimationcited by 0
  5. 2025Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Videocited by 0
  6. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0