mlsanityData to 7 Oct 2026

Towards visually prompted keyword localisation for zero-resource spoken languages

arXiv 2210.06229 ↗First posted 12 Oct 2022cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Imagine being able to show a system a visual depiction of a keyword and finding spoken utterances that contain this keyword from a zero-resource speech corpus. We formalise this task and call it visually prompted keyword localisation (VPKL): given an image of a keyword, detect and predict where in an utterance the keyword occurs. To do VPKL, we propose a speech-vision model with a novel localising attention mechanism which we train with a new keyword sampling scheme. We show that these innovations give improvements in VPKL over an existing speech-vision model. We also compare to a visual bag-of-words (BoW) model where images are automatically tagged with visual labels and paired with unlabelled speech. Although this visual BoW can be queried directly with a written keyword (while our's takes image queries), our new model still outperforms the visual BoW in both detection and localisation, giving a 16% relative improvement in localisation F1.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

25 of 39 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  5. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  6. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  7. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  8. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  9. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  10. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  11. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  12. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
13 more
  1. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  2. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  3. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  4. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  5. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  6. 2018Multimodal One-Shot Learning of Speech and Imagescited by 11
  7. 2019A computational model of early language acquisition from audiovisual experiences of young infantscited by 10
  8. 2016From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learningcited by 8
  9. 2019On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrievalcited by 8
  10. 2020Unsupervised vs. transfer learning for multimodal one-shot matching of speech and imagescited by 7
  11. 2020Towards localisation of keywords in speech using weak supervisioncited by 7
  12. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  13. 2020Direct multimodal few-shot learning of speech and imagescited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4.

6 papers.

  1. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  2. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  3. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2
  4. 2023Visually grounded few-shot word acquisition with fewer shotscited by 1
  5. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  6. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0