mlsanityData to 7 Oct 2026

Supervising Sound Localization by In-the-wild Egomotion

arXiv 2610.01388 ↗First posted 1 Oct 2026cs.CV cs.AI cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a method for learning binaural sound localization using egomotion as a supervisory signal. Over the course of a video, the cameras direction to a sound source will change as the camera moves. We train an audio model to predict sound directions that are consistent with visual estimates of camera motion, which we obtain using traditional methods from multi-view geometry. This provides a weak but plentiful form of supervision that we combine with traditional binaural cues. To evaluate this method, we propose a dataset of real-world audio-visual videos with egomotion. We show that our model can successfully learn from real-world data and that it performs well on sound localization tasks

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

49 of 66 references.

References · 49

  1. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  2. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  3. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  4. 2017Look, Listen and Learncited by 452
  5. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  6. 2016FMA: A Dataset For Music Analysiscited by 205
  7. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  8. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  9. 2021Localizing Visual Sounds the Hard Waycited by 169
  10. 2022Audio-Visual Segmentationcited by 148
  11. 2019The Sound of Motionscited by 145
  12. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
37 more
  1. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  2. 2024Synchformer: Efficient Synchronization from Sparse Cuescited by 94
  3. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  4. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  5. 2018Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networkscited by 84
  6. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  7. 2022Perspective Fields for Single Image Camera Calibrationcited by 75
  8. 20182.5D Visual Soundcited by 63
  9. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  10. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  11. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  12. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  13. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  14. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  15. 2021Audio-Visual Synchronisation in the wildcited by 39
  16. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  17. 2024360+x: A Panoptic Multi-modal Scene Understanding Datasetcited by 35
  18. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 34
  19. 2023Novel-View Acoustic Synthesiscited by 33
  20. 2021Beyond Image to Depth: Improving Depth Prediction using Echoescited by 24
  21. 2020A Dataset of Reverberant Spatial Sound Scenes with Moving Sources for Sound Event Localization and Detectioncited by 21
  22. 2021A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detectioncited by 21
  23. 2021A Dataset of Dynamic Reverberant Sound Scenes with Directional Interferers for Sound Event Localization and Detectioncited by 20
  24. 2019A multi-room reverberant dataset for sound event localization and detectioncited by 18
  25. 2022Sound Localization by Self-Supervised Time Delay Estimationcited by 18
  26. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  27. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  28. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  29. 2019The LOCATA Challenge: Acoustic Source Localization and Trackingcited by 16
  30. 2022STARSS22: A dataset of spatial recordings of real scenes with spatiotemporal annotations of sound eventscited by 16
  31. 2024FAR: Flexible, Accurate and Robust 6DoF Relative Camera Pose Estimationcited by 12
  32. 2022L3DAS22 Challenge: Learning 3D Audio Sources in a Real Office Environmentcited by 10
  33. 2021L3DAS21 Challenge: Machine Learning for 3D Audio Signal Processingcited by 9
  34. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  35. 2020BatVision with GCC-PHAT Features for Better Sound to Vision Predictionscited by 5
  36. 2024Spherical World-Locking for Audio-Visual Localization in Egocentric Videoscited by 5
  37. 2022Wearable SELD dataset: Dataset for sound event localization and detection using wearable devices around headcited by 2

Built on by

Not cited yet.