# A Critical Assessment of Visual Sound Source Localization Models Including Negative Audio

arXiv: https://arxiv.org/abs/2410.01020 · HTML: https://mlsanity.com/paper/2410.01020/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Multiple Sound Sources Localization from Coarse to Fine](https://mlsanity.com/paper/2007.06355/) (2020)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching](https://mlsanity.com/paper/2010.05466/) (2020)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [Audio-Visual Grouping Network for Sound Localization from Mixtures](https://mlsanity.com/paper/2303.17056/) (2023)
- [Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning](https://mlsanity.com/paper/2303.11302/) (2023)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds](https://mlsanity.com/paper/2111.14843/) (2021)
- [Visual Sound Localization in the Wild by Cross-Modal Interference Erasing](https://mlsanity.com/paper/2202.06406/) (2022)
- [T-VSL: Text-Guided Visual Sound Source Localization in Mixtures](https://mlsanity.com/paper/2404.01751/) (2024)
- [Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge](https://mlsanity.com/paper/2403.17420/) (2024)
- [Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment](https://mlsanity.com/paper/2407.13676/) (2024)
- [How to Listen? Rethinking Visual Sound Localization](https://mlsanity.com/paper/2204.05156/) (2022)

## Built on by (most cited first, 4 of 4)

- [Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?](https://mlsanity.com/paper/2502.00358/) (2025)
- [From Waveforms to Pixels: A Survey on Audio-Visual Segmentation](https://mlsanity.com/paper/2508.03724/) (2025)
- [TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization](https://mlsanity.com/paper/2412.01488/) (2024)
- [Learning from Silence and Noise for Visual Sound Source Localization](https://mlsanity.com/paper/2508.21761/) (2025)

## Abstract (arXiv)

The task of Visual Sound Source Localization (VSSL) involves identifying the location of sound sources in visual scenes, integrating audio-visual data for enhanced scene understanding. Despite advancements in state-of-the-art (SOTA) models, we observe three critical flaws: i) The evaluation of the models is mainly focused in sounds produced by objects that are visible in the image, ii) The evaluation often assumes a prior knowledge of the size of the sounding object, and iii) No universal threshold for localization in real-world scenarios is established, as previous approaches only consider positive examples without accounting for both positive and negative cases. In this paper, we introduce a novel test set and metrics designed to complete the current standard evaluation of VSSL models by testing them in scenarios where none of the objects in the image corresponds to the audio input, i.e. a negative audio. We consider three types of negative audio: silence, noise and offscreen. Our analysis reveals that numerous SOTA models fail to appropriately adjust their predictions based on audio input, suggesting that these models may not be leveraging audio information as intended. Additionally, we provide a comprehensive analysis of the range of maximum values in the estimated audio-visual similarity maps, in both positive and negative audio cases, and show that most of the models are not discriminative enough, making them unfit to choose a universal threshold appropriate to perform sound localization without any a priori information of the sounding object, that is, object size and visibility.