# Learning to Recognise Words using Visually Grounded Speech

arXiv: https://arxiv.org/abs/2006.00512 · HTML: https://mlsanity.com/paper/2006.00512/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2020.

## Built on by (most cited first, 7 of 7)

- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation](https://mlsanity.com/paper/2109.14200/) (2021)
- [ConceptBeam: Concept Driven Target Speech Extraction](https://mlsanity.com/paper/2207.11964/) (2022)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [Modelling word learning and recognition using visually grounded speech](https://mlsanity.com/paper/2203.06937/) (2022)
- [Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings](https://mlsanity.com/paper/2409.06013/) (2024)

## Abstract (arXiv)

We investigated word recognition in a Visually Grounded Speech model. The model has been trained on pairs of images and spoken captions to create visually grounded embeddings which can be used for speech to image retrieval and vice versa. We investigate whether such a model can be used to recognise words by embedding isolated words and using them to retrieve images of their visual referents. We investigate the time-course of word recognition using a gating paradigm and perform a statistical analysis to see whether well known word competition effects in human speech processing influence word recognition. Our experiments show that the model is able to recognise words, and the gating paradigm reveals that words can be recognised from partial input as well and that recognition is negatively influenced by word competition from the word initial cohort.