mlsanityData to 7 Oct 2026

Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings

arXiv 2409.06013 ↗First posted 9 Sep 2024cs.CL cs.CV eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 9 Sep 2024.

Builds on

19 of 24 references.

References · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  5. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  6. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  7. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  8. 2017Representations of language in a model of visually grounded speech signalcited by 37
  9. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  10. 2022BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpuscited by 31
  11. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  12. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
7 more
  1. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  2. 2022Building African Voicescited by 14
  3. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  4. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  5. 2020Learning to Recognise Words using Visually Grounded Speechcited by 7
  6. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  7. 2023Visually grounded few-shot word learning in low-resource settingscited by 4

Built on by

Not cited yet.