mlsanityData to 7 Oct 2026

Sound Localization by Self-Supervised Time Delay Estimation

arXiv 2204.12489 ↗First posted 26 Apr 2022cs.CV cs.SD eess.AScited by 18

Not on the map. Why?

Abstract (arXiv)

Sounds reach one microphone in a stereo pair sooner than the other, resulting in an interaural time delay that conveys their directions. Estimating a sound's time delay requires finding correspondences between the signals recorded by each microphone. We propose to learn these correspondences through self-supervision, drawing on recent techniques from visual tracking. We adapt the contrastive random walk of Jabri et al. to learn a cycle-consistent representation from unlabeled stereo sounds, resulting in a model that performs on par with supervised methods on "in the wild" internet recordings. We also propose a multimodal contrastive learning model that solves a visually-guided localization task: estimating the time delay for a particular person in a multi-speaker mixture, given a visual representation of their face. Project site: https://ificl.github.io/stereocrw/

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 51 of its 89 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

51 of 89 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  9. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  10. 2020Tracking Objects as Pointscited by 483
  11. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  12. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
38 more
  1. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  2. 2018The Sound of Pixelscited by 258
  3. 2017Objects that Soundcited by 254
  4. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  5. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  6. 2016FMA: A Dataset For Music Analysiscited by 205
  7. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  8. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  9. 2018Tracking Emerges by Colorizing Videoscited by 166
  10. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  11. 2017Pyroomacoustics: A Python package for audio room simulations and array processing algorithmscited by 131
  12. 2020What Matters in Unsupervised Optical Flowcited by 124
  13. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  14. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  15. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  16. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  17. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  18. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63
  19. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  20. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  21. 2020Watching the World Go By: Representation Learning from Unlabeled Videoscited by 59
  22. 2021Towards Learning Universal Audio Representationscited by 51
  23. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  24. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  25. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  26. 2021Representation Learning via Global Temporal Alignment and Cycle-Consistencycited by 43
  27. 2021Multi-Format Contrastive Learning of Audio Representationscited by 39
  28. 2019Machine learning in acoustics: theory and applicationscited by 36
  29. 2022Learning Pixel Trajectories with Multiscale Contrastive Random Walkscited by 36
  30. 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
  31. 2017Direction of arrival estimation for multiple sound sources using convolutional recurrent neural networkcited by 27
  32. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  33. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  34. 2017Broadband DOA estimation using Convolutional neural networks trained with noise signalscited by 15
  35. 2020Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networkscited by 15
  36. 2019End-to-end Binaural Sound Localisation from the Raw Waveformcited by 6
  37. 2020BatVision with GCC-PHAT Features for Better Sound to Vision Predictionscited by 5
  38. 2020Time Difference of Arrival Estimation from Frequency-Sliding Generalized Cross-Correlations Using Convolutional Neural Networkscited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 6; 2025: 2; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  2. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  5. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  6. 2024360+x: A Panoptic Multi-modal Scene Understanding Datasetcited by 35
  7. 2023Audio-Visual Class-Incremental Learningcited by 33
  8. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  9. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  10. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  11. 2024Hearing Anything Anywherecited by 12
  12. 2025Guiding Audio Editing with Audio Language Modelcited by 11
6 more
  1. 2023RealImpact: A Dataset of Impact Sound Fields for Real Objectscited by 8
  2. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  3. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  4. 2023Exploring Self-Supervised Contrastive Learning of Spatial Sound Event Representationcited by 2
  5. 2023The Un-Kidnappable Robot: Acoustic Localization of Sneaking Peoplecited by 2
  6. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0