mlsanityData to 7 Oct 2026

Learning to Localize Sound Source in Visual Scenes

arXiv 1803.03849 ↗First posted 10 Mar 2018cs.CV cs.AI cs.MMcited by 188

Not on the map. Why?

Abstract (arXiv)

Visual events are usually accompanied by sounds in our daily lives. We pose the question: Can the machine learn the correspondence between visual scene and the sound, and localize the sound source only by observing sound and visual scene pairs like human? In this paper, we propose a novel unsupervised algorithm to address the problem of localizing the sound source in visual scenes. A two-stream network structure which handles each modality, with attention mechanism is developed for sound source localization. Moreover, although our network is formulated within the unsupervised learning framework, it can be extended to a unified architecture with a simple modification for the supervised and semi-supervised learning settings as well. Meanwhile, a new sound source dataset is developed for performance evaluation. Our empirical evaluation shows that the unsupervised method eventually go through false conclusion in some cases. We show that even with a few supervision, false conclusion is able to be corrected and the source of sound in a visual scene can be localized effectively.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 37; 2022: 32; 2023: 44; 2024: 37; 2025: 18; 2026: 20.

188 papers.

Sort:

By area:

  1. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  2. 2021Localizing Visual Sounds the Hard Waycited by 169
  3. 2022Audio-Visual Segmentationcited by 148
  4. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  5. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  6. 2022Localizing Visual Sounds the Easy Waycited by 81
  7. 2021Robust Audio-Visual Instance Discriminationcited by 80
  8. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 73
  9. 2023Audio-Visual Segmentation with Semanticscited by 73
  10. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 72
  11. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  12. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
38 more
  1. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 66
  2. 2022Learning Neural Acoustic Fieldscited by 65
  3. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  4. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  5. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  6. 2023AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentationcited by 52
  7. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  8. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 46
  9. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  10. 2023Annotation-free Audio-Visual Segmentationcited by 44
  11. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 43
  12. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  13. 2023Egocentric Audio-Visual Object Localizationcited by 40
  14. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  15. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  16. 2023Connecting Multi-modal Contrastive Representationscited by 36
  17. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  18. 2023Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizercited by 35
  19. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  20. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  21. 2023Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentationcited by 33
  22. 2023Audio-Visual Class-Incremental Learningcited by 33
  23. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  24. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  25. 2023Audio-aware Query-enhanced Transformer for Audio-Visual Segmentationcited by 32
  26. 2023Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semanticscited by 32
  27. 2024Ref-AVS: Refer and Segment Objects in Audio-Visual Scenescited by 32
  28. 2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
  29. 2023BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledgecited by 30
  30. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  31. 2024Scaling 4D Representationscited by 30
  32. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  33. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  34. 2022Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsingcited by 29
  35. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  36. 2023Progressive Spatio-temporal Perception for Audio-Visual Question Answeringcited by 29
  37. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  38. 2024Text-to-Audio Generation Synchronized with Videoscited by 28

50 most cited of 188 All 188 (JSON)