mlsanityData to 14 Oct 2026

Sound Source Localization is All about Cross-Modal Alignment

arXiv 2309.10724 ↗First posted 19 Sep 2023cs.CV cs.AI cs.MMcited by 31

Not on the map. Why?

Abstract (arXiv)

Humans can easily perceive the direction of sound sources in a visual scene, termed sound source localization. Recent studies on learning-based sound source localization have mainly explored the problem from a localization perspective. However, prior arts and existing benchmarks do not account for a more important aspect of the problem, cross-modal semantic understanding, which is essential for genuine sound source localization. Cross-modal semantic understanding is important in understanding semantically mismatched audio-visual events, e.g., silent objects, or off-screen sounds. To account for this, we propose a cross-modal alignment task as a joint task with sound source localization to better learn the interaction between audio and visual modalities. Thereby, we achieve high localization performance with strong cross-modal semantic understanding. Our method outperforms the state-of-the-art approaches in both sound source localization and cross-modal retrieval. Our work suggests that jointly tackling both tasks is necessary to conquer genuine sound source localization.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 53 of its 67 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

53 of 67 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  9. 2017Look, Listen and Learncited by 458
  10. 2014Deep metric learning using Triplet networkcited by 422
  11. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  12. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
38 more
  1. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  2. 2017Objects that Soundcited by 256
  3. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  5. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  6. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  7. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  8. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  9. 2021Localizing Visual Sounds the Hard Waycited by 170
  10. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  11. 2022Audio-Visual Segmentationcited by 151
  12. 2019The Sound of Motionscited by 146
  13. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  14. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  15. 2020Music Gesture for Visual Sound Separationcited by 112
  16. 2019Co-Separating Sounds of Visual Objectscited by 110
  17. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  18. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  19. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 96
  20. 2022Localizing Visual Sounds the Easy Waycited by 82
  21. 2021Robust Audio-Visual Instance Discriminationcited by 80
  22. 2016Ambient Sound Provides Supervision for Visual Learningcited by 78
  23. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  24. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  25. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  26. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  27. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  28. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 44
  29. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  30. 2021Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Predictioncited by 36
  31. 2022Learning Sound Localization Better From Semantically Similar Samplescited by 32
  32. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  33. 2019Learning to Localize Sound Sources in Visual Scenes: Analysis and Applicationscited by 31
  34. 2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
  35. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  36. 2020Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervisioncited by 19
  37. 2022MarginNCE: Robust Sound Localization with a Negative Margincited by 16
  38. 2020Do We Need Sound for Sound Source Localization?cited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 12; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  2. 2023Audio-Visual Instance Segmentationcited by 21
  3. 2024EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Drivingcited by 19
  4. 2024T-VSL: Text-Guided Visual Sound Source Localization in Mixturescited by 19
  5. 2025Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localizationcited by 18
  6. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  7. 2023Can CLIP Help Sound Source Localization?cited by 14
  8. 2024Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledgecited by 14
  9. 2024CPM: Class-conditional Prompting Machine for Audio-visual Segmentationcited by 11
  10. 2024Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understandingcited by 7
  11. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  12. 2025Object-aware Sound Source Localization via Audio-Visual Scene Understandingcited by 5
19 more
  1. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 5
  2. 2025Improving Sound Source Localization with Joint Slot Attention on Image and Audiocited by 4
  3. 2025What's Making That Sound Right Now? Video-centric Audio-Visual Localizationcited by 4
  4. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3
  5. 2025UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsingcited by 2
  6. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 2
  7. 2026Seeing Through Touch: Tactile-Driven Visual Localization of Material Regionscited by 2
  8. 2024TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorizationcited by 1
  9. 2025Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localizationcited by 1
  10. 2025Layover or Direct Flight: Rethinking Audio-Guided Image Segmentationcited by 1
  11. 2026SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentationcited by 1
  12. 2026See & Sniff: Learning Visuo-Olfactory Representationscited by 1
  13. 2025Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixupcited by 0
  14. 2025Clink! Chop! Thud! -- Learning Object Sounds from Real-World Interactionscited by 0
  15. 2025Segmenting Collision Sound Sources in Egocentric Videoscited by 0
  16. 2026SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contextscited by 0
  17. 2026Unlocking Spatial Grounding in Large Audio-Visual Retrieval modelscited by 0
  18. 2026Binaural Audio-Visual Instance Segmentationcited by 0
  19. 2026SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attentioncited by 0