Visually grounded few-shot word learning in low-resource settings
Not on the map. Why?
Abstract (arXiv)
We propose a visually grounded speech model that learns new words and their visual depictions from just a few word-image example pairs. Given a set of test images and a spoken query, we ask the model which image depicts the query word. Previous work has simplified this few-shot learning problem by either using an artificial setting with digit word-image pairs or by using a large number of examples per class. Moreover, all previous studies were performed using English speech-image data. We propose an approach that can work on natural word-image pairs but with less examples, i.e. fewer shots, and then illustrate how this approach can be applied for multimodal few-shot learning in a real low-resource language, Yor`ub'a. Our approach involves using the given word-image example pairs to mine new unsupervised word-image training pairs from large collections of unlabelled speech and images. Additionally, we use a word-to-image attention mechanism to determine word-image similarity. With this new model, we achieve better performance with fewer shots than previous approaches on an existing English benchmark. Many of the model's mistakes are due to confusion between visual concepts co-occurring in similar contexts. The experiments on Yor`ub'a show the benefit of transferring knowledge from a multimodal model trained on a larger set of English speech-image data.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 20 Jun 2023.
Builds on
27 of 47 references.
References · 27
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
- 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
- 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
- 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
- 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
- 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
- 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
- 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
15 more
- 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
- 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
- 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
- 2018Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detectioncited by 17
- 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
- 2018Multimodal One-Shot Learning of Speech and Imagescited by 11
- 2019Semantic query-by-example speech search using visual groundingcited by 11
- 2020Improved acoustic word embeddings for zero-resource languages using multilingual transfercited by 11
- 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
- 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
- 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
- 2020Unsupervised vs. transfer learning for multimodal one-shot matching of speech and imagescited by 7
- 2020Direct multimodal few-shot learning of speech and imagescited by 6
- 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
- 2023Visually grounded few-shot word acquisition with fewer shotscited by 1
Built on by
Citing papers per year: 2024: 3; 2025: 1.
4 papers.