Visually grounded few-shot word acquisition with fewer shots
Not on the map. Why?
Abstract (arXiv)
We propose a visually grounded speech model that acquires new words and their visual depictions from just a few word-image example pairs. Given a set of test images and a spoken query, we ask the model which image depicts the query word. Previous work has simplified this problem by either using an artificial setting with digit word-image pairs or by using a large number of examples per class. We propose an approach that can work on natural word-image pairs but with less examples, i.e. fewer shots. Our approach involves using the given word-image example pairs to mine new unsupervised word-image training pairs from large collections of unlabelled speech and images. Additionally, we use a word-to-image attention mechanism to determine word-image similarity. With this new model, we achieve better performance with fewer shots than any existing approach.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 25 May 2023.
Builds on
19 of 34 references.
References · 19
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
- 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
- 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
- 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
- 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
- 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
7 more
- 2018Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detectioncited by 17
- 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
- 2018Multimodal One-Shot Learning of Speech and Imagescited by 11
- 2019Semantic query-by-example speech search using visual groundingcited by 11
- 2020Unsupervised vs. transfer learning for multimodal one-shot matching of speech and imagescited by 7
- 2020Direct multimodal few-shot learning of speech and imagescited by 6
- 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
Built on by
Citing papers per year: 2023: 1.
1 paper.