# Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech

arXiv: https://arxiv.org/abs/1909.08491 · HTML: https://mlsanity.com/paper/1909.08491/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2019.

## Built on by (most cited first, 13 of 13)

- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Word Discovery in Visually Grounded, Self-Supervised Speech Models](https://mlsanity.com/paper/2203.15081/) (2022)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval](https://mlsanity.com/paper/2104.01894/) (2021)
- [Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation](https://mlsanity.com/paper/2109.14200/) (2021)
- [Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model](https://mlsanity.com/paper/2305.11435/) (2023)
- [Learning English with Peppa Pig](https://mlsanity.com/paper/2202.12917/) (2022)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 edition](https://mlsanity.com/paper/2107.06546/) (2021)
- [Semantic sentence similarity: size does not always matter](https://mlsanity.com/paper/2106.08648/) (2021)
- [A model of early word acquisition based on realistic-scale audiovisual naming events](https://mlsanity.com/paper/2406.05259/) (2024)
- [Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priors](https://mlsanity.com/paper/2603.08359/) (2026)
- [Modelling word learning and recognition using visually grounded speech](https://mlsanity.com/paper/2203.06937/) (2022)

## Abstract (arXiv)

In this paper, we study how word-like units are represented and activated in a recurrent neural model of visually grounded speech. The model used in our experiments is trained to project an image and its spoken description in a common representation space. We show that a recurrent model trained on spoken sentences implicitly segments its input into word-like units and reliably maps them to their correct visual referents. We introduce a methodology originating from linguistics to analyse the representation learned by neural networks -- the gating paradigm -- and show that the correct representation of a word is only activated if the network has access to first phoneme of the target word, suggesting that the network does not rely on a global acoustic pattern. Furthermore, we find out that not all speech frames (MFCC vectors in our case) play an equal role in the final encoded representation of a given word, but that some frames have a crucial effect on it. Finally, we suggest that word representation could be activated through a process of lexical competition.