# Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

arXiv: https://arxiv.org/abs/2607.24786 · HTML: https://mlsanity.com/paper/2607.24786/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 31 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation](https://mlsanity.com/paper/2305.01836/) (2023)
- [FLAIR: VLM with Fine-grained Language-informed Image Representations](https://mlsanity.com/paper/2412.03561/) (2024)
- [Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learning](https://mlsanity.com/paper/2303.11302/) (2023)
- [Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation](https://mlsanity.com/paper/2206.12772/) (2022)
- [Sound Source Localization is All about Cross-Modal Alignment](https://mlsanity.com/paper/2309.10724/) (2023)
- [Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language](https://mlsanity.com/paper/2406.05629/) (2024)
- [Transfer Learning from Audio-Visual Grounding to Speech Recognition](https://mlsanity.com/paper/1907.04355/) (2019)
- [MarginNCE: Robust Sound Localization with a Negative Margin](https://mlsanity.com/paper/2211.01966/) (2022)
- [Open-Vocabulary Audio-Visual Semantic Segmentation](https://mlsanity.com/paper/2407.21721/) (2024)
- [CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment](https://mlsanity.com/paper/2505.01237/) (2025)
- [Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning](https://mlsanity.com/paper/2512.19687/) (2025)
- [What's Making That Sound Right Now? Video-centric Audio-Visual Localization](https://mlsanity.com/paper/2507.04667/) (2025)
- [TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization](https://mlsanity.com/paper/2412.01488/) (2024)
- [Audio Visual Segmentation Through Text Embeddings](https://mlsanity.com/paper/2502.16359/) (2025)

## Abstract (arXiv)

Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challenging, as models must locate sound sources from temporally aligned audio-visual data without pixel-level supervision. Recent large-scale audio-visual retrieval models, trained at unprecedented scale, encode rich multimodal structure. We show their latent representations, though optimized for global alignment, can nonetheless enable fine-grained spatial grounding. While spatial detail is progressively lost in the upper layers of retrieval backbones due to global pooling, intermediate visual tokens retain highly structured spatial information. To exploit this, we introduce LAIP (Localization via Audio-Informed Pooling), a framework that employs a lightweight Audio-informed Spatial Pooling (AiSP) to replace the standard global aggregation module. By querying intermediate visual tokens with audio aligned at the frame level, LAIP recovers localized spatial information that is otherwise discarded by the retrieval pipeline, with the largest gains observed for PE-AV, a stack with underlying temporal aggregation. Our approach achieves state-of-the-art performance on AVSBench and AVATAR, nearly doubling previous results on the latter, improving average CIoU from 13.21 to 26.22.