# Unsupervised Sound Localization via Iterative Contrastive Learning

arXiv: https://arxiv.org/abs/2104.00315 · HTML: https://mlsanity.com/paper/2104.00315/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 59 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2007.10558/) (2020)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Labelling unlabelled videos from scratch with multi-modal self-supervision](https://mlsanity.com/paper/2006.13662/) (2020)
- [Learning Representations from Audio-Visual Spatial Alignment](https://mlsanity.com/paper/2011.01819/) (2020)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Dual-modality seq2seq network for audio-visual event localization](https://mlsanity.com/paper/1902.07473/) (2019)
- [Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1712.07271/) (2017)

## Built on by (most cited first, 28 of 28)

- [Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning](https://mlsanity.com/paper/2303.11302/) (2023)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation](https://mlsanity.com/paper/2307.13236/) (2023)
- [Learning Sound Localization Better From Semantically Similar Samples](https://mlsanity.com/paper/2202.03007/) (2022)
- [Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes](https://mlsanity.com/paper/2203.13412/) (2022)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [Audio-Visual Segmentation via Unlabeled Frame Exploitation](https://mlsanity.com/paper/2403.11074/) (2024)
- [AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation](https://mlsanity.com/paper/2207.10141/) (2022)
- [Visual Sound Localization in the Wild by Cross-Modal Interference Erasing](https://mlsanity.com/paper/2202.06406/) (2022)
- [Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization](https://mlsanity.com/paper/2505.11217/) (2025)
- [Can CLIP Help Sound Source Localization?](https://mlsanity.com/paper/2311.04066/) (2023)
- [Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge](https://mlsanity.com/paper/2403.17420/) (2024)
- [A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!](https://mlsanity.com/paper/2206.13390/) (2022)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [Estimating Visual Information From Audio Through Manifold Learning](https://mlsanity.com/paper/2208.02337/) (2022)
- [Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization](https://mlsanity.com/paper/2210.05242/) (2022)
- [Contrastive Learning of Global-Local Video Representations](https://mlsanity.com/paper/2104.05418/) (2021)
- [Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention](https://mlsanity.com/paper/2106.09669/) (2021)
- [Object-aware Sound Source Localization via Audio-Visual Scene Understanding](https://mlsanity.com/paper/2506.18557/) (2025)
- [SAVE: Segment Audio-Visual Easy way using Segment Anything Model](https://mlsanity.com/paper/2407.02004/) (2024)
- [Enhancing Sound Source Localization via False Negative Elimination](https://mlsanity.com/paper/2408.16448/) (2024)
- [Unveiling Visual Biases in Audio-Visual Localization Benchmarks](https://mlsanity.com/paper/2409.06709/) (2024)
- [TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization](https://mlsanity.com/paper/2412.01488/) (2024)
- [Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization](https://mlsanity.com/paper/2505.05343/) (2025)
- [Action Dubber: Timing Audible Actions via Inflectional Flow](https://mlsanity.com/paper/2506.13320/) (2025)
- [SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts](https://mlsanity.com/paper/2603.22732/) (2026)
- [Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning](https://mlsanity.com/paper/2604.06824/) (2026)

## Abstract (arXiv)

Sound localization aims to find the source of the audio signal in the visual scene. However, it is labor-intensive to annotate the correlations between the signals sampled from the audio and visual modalities, thus making it difficult to supervise the learning of a machine for this task. In this work, we propose an iterative contrastive learning framework that requires no data annotations. At each iteration, the proposed method takes the 1) localization results in images predicted in the previous iteration, and 2) semantic relationships inferred from the audio signals as the pseudo-labels. We then use the pseudo-labels to learn the correlation between the visual and audio signals sampled from the same video (intra-frame sampling) as well as the association between those extracted across videos (inter-frame relation). Our iterative strategy gradually encourages the localization of the sounding objects and reduces the correlation between the non-sounding regions and the reference audio. Quantitative and qualitative experimental results demonstrate that the proposed framework performs favorably against existing unsupervised and weakly-supervised methods on the sound localization task.