# MarginNCE: Robust Sound Localization with a Negative Margin

arXiv: https://arxiv.org/abs/2211.01966 · HTML: https://mlsanity.com/paper/2211.01966/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 30 of 31)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019)
- [Audiovisual SlowFast Networks for Video Recognition](https://mlsanity.com/paper/2001.08740/) (2020)
- [Negative Margin Matters: Understanding Margin in Few-shot Classification](https://mlsanity.com/paper/2003.12060/) (2020)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Self-Supervised Generation of Spatial Audio for 360 Video](https://mlsanity.com/paper/1809.02587/) (2018)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds](https://mlsanity.com/paper/2011.01143/) (2020)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Telling Left from Right: Learning Spatial Correspondence of Sight and Sound](https://mlsanity.com/paper/2006.06175/) (2020)
- [Delving into Inter-Image Invariance for Unsupervised Visual Representations](https://mlsanity.com/paper/2008.11702/) (2020)
- [Learning Sound Localization Better From Semantically Similar Samples](https://mlsanity.com/paper/2202.03007/) (2022)
- [Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications](https://mlsanity.com/paper/1911.09649/) (2019)
- [Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes](https://mlsanity.com/paper/2203.13412/) (2022)

## Built on by (most cited first, 16 of 16)

- [Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment](https://mlsanity.com/paper/2303.17490/) (2023)
- [Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer](https://mlsanity.com/paper/2309.07929/) (2023)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time](https://mlsanity.com/paper/2407.01851/) (2024)
- [T-VSL: Text-Guided Visual Sound Source Localization in Mixtures](https://mlsanity.com/paper/2404.01751/) (2024)
- [Can CLIP Help Sound Source Localization?](https://mlsanity.com/paper/2311.04066/) (2023)
- [Can Textual Semantics Mitigate Sounding Object Segmentation Preference?](https://mlsanity.com/paper/2407.10947/) (2024)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment](https://mlsanity.com/paper/2412.06209/) (2024)
- [Cross Pseudo-Labeling for Semi-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2403.03095/) (2024)
- [TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization](https://mlsanity.com/paper/2412.01488/) (2024)
- [Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models](https://mlsanity.com/paper/2506.06537/) (2025)
- [Learning from Silence and Noise for Visual Sound Source Localization](https://mlsanity.com/paper/2508.21761/) (2025)
- [Acoustic Field Video for Multimodal Scene Understanding](https://mlsanity.com/paper/2601.17123/) (2026)
- [SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts](https://mlsanity.com/paper/2603.22732/) (2026)
- [Unlocking Spatial Grounding in Large Audio-Visual Retrieval models](https://mlsanity.com/paper/2607.24786/) (2026)

## Abstract (arXiv)

The goal of this work is to localize sound sources in visual scenes with a self-supervised approach. Contrastive learning in the context of sound source localization leverages the natural correspondence between audio and visual signals where the audio-visual pairs from the same source are assumed as positive, while randomly selected pairs are negatives. However, this approach brings in noisy correspondences; for example, positive audio and visual pair signals that may be unrelated to each other, or negative pairs that may contain semantically similar samples to the positive one. Our key contribution in this work is to show that using a less strict decision boundary in contrastive learning can alleviate the effect of noisy correspondences in sound source localization. We propose a simple yet effective approach by slightly modifying the contrastive loss with a negative margin. Extensive experimental results show that our approach gives on-par or better performance than the state-of-the-art methods. Furthermore, we demonstrate that the introduction of a negative margin to existing methods results in a consistent improvement in performance.