VeS: Teaching Pixels to Listen Without Supervision
Not on the map. Why?
Abstract (arXiv)
Recent dense audio-visual (AV) models achieve impressive retrieval and emergent localization, but almost all evidence comes from English-centric, caption-rich web video. It is unclear whether these objectives survive in low-resource, code-switched, and noisy multilingual settings that typify developing regions. We show they do**-**and that the choice of aggregation function becomes even more critical. Using a multilingual subset of Project Vaani spanning dozens of Indian languages and dialectal variants, we compare three contrastive objectives: (i) a global mean-pooled loss (CLIP-style), (ii) a dense max-mean token matcher (DenseAV-style), and (iii) a simple hybrid (motivated by frozen-vision alignment strategies). The dense objective delivers a +59% relative R@1 (Audio Visual) improvement over global pooling and substantially lower mean/median ranks, while consistently producing sharp zero-shot localization heatmaps of spoken objects-despite keeping the vision backbone entirely frozen (no LoRA / partial fine-tuning). Our results demonstrate that dense token routing is not a luxury of high-resource English corpora; it is more decisive when annotations and acoustic cleanliness are scarce. We release the codebase and trained models.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 8 of its 9 references that are in the map. First posted on arXiv 29 Jul 2025.
Builds on
8 of 9 references.
References · 8
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
- 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
- 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 573
- 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 130
- 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
- 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
Built on by
Not cited yet.