# Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes

arXiv: https://arxiv.org/abs/2203.13412 · HTML: https://mlsanity.com/paper/2203.13412/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 30 of 42)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)

## Built on by (most cited first, 30 of 31)

- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Egocentric Audio-Visual Object Localization](https://mlsanity.com/paper/2303.13471/) (2023)
- [Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation](https://mlsanity.com/paper/2307.13236/) (2023)
- [Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics](https://mlsanity.com/paper/2307.16620/) (2023)
- [BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge](https://mlsanity.com/paper/2308.10175/) (2023)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [Audio-Visual Segmentation via Unlabeled Frame Exploitation](https://mlsanity.com/paper/2403.11074/) (2024)
- [Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time](https://mlsanity.com/paper/2407.01851/) (2024)
- [Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2407.11820/) (2024)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [MarginNCE: Robust Sound Localization with a Negative Margin](https://mlsanity.com/paper/2211.01966/) (2022)
- [Hear The Flow: Optical Flow-Based Self-Supervised Visual Sound Source Localization](https://mlsanity.com/paper/2211.03019/) (2022)
- [Can CLIP Help Sound Source Localization?](https://mlsanity.com/paper/2311.04066/) (2023)
- [Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge](https://mlsanity.com/paper/2403.17420/) (2024)
- [Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization](https://mlsanity.com/paper/2403.03145/) (2024)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization](https://mlsanity.com/paper/2210.05242/) (2022)
- [Object-aware Sound Source Localization via Audio-Visual Scene Understanding](https://mlsanity.com/paper/2506.18557/) (2025)
- [SAVE: Segment Audio-Visual Easy way using Segment Anything Model](https://mlsanity.com/paper/2407.02004/) (2024)
- [Improving Sound Source Localization with Joint Slot Attention on Image and Audio](https://mlsanity.com/paper/2504.15118/) (2025)
- [Enhancing Sound Source Localization via False Negative Elimination](https://mlsanity.com/paper/2408.16448/) (2024)
- [Unveiling Visual Biases in Audio-Visual Localization Benchmarks](https://mlsanity.com/paper/2409.06709/) (2024)
- [Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding](https://mlsanity.com/paper/2306.10684/) (2023)
- [Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization](https://mlsanity.com/paper/2308.04767/) (2023)
- [Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization](https://mlsanity.com/paper/2505.05343/) (2025)
- [SemiPL: A Semi-supervised Method for Event Sound Source Localization](https://mlsanity.com/paper/2404.19615/) (2024)
- [SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts](https://mlsanity.com/paper/2603.22732/) (2026)
- [Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning](https://mlsanity.com/paper/2604.06824/) (2026)

## Abstract (arXiv)

Sound source localization in visual scenes aims to localize objects emitting the sound in a given image. Recent works showing impressive localization performance typically rely on the contrastive learning framework. However, the random sampling of negatives, as commonly adopted in these methods, can result in misalignment between audio and visual features and thus inducing ambiguity in localization. In this paper, instead of following previous literature, we propose Self-Supervised Predictive Learning (SSPL), a negative-free method for sound localization via explicit positive mining. Specifically, we first devise a three-stream network to elegantly associate sound source with two augmented views of one corresponding video frame, leading to semantically coherent similarities between audio and visual features. Second, we introduce a novel predictive coding module for audio-visual feature alignment. Such a module assists SSPL to focus on target objects in a progressive manner and effectively lowers the positive-pair learning difficulty. Experiments show surprising results that SSPL outperforms the state-of-the-art approach on two standard sound localization benchmarks. In particular, SSPL achieves significant improvements of 8.6% cIoU and 3.4% AUC on SoundNet-Flickr compared to the previous best. Code is available at: https://github.com/zjsong/SSPL.