# Modelling word learning and recognition using visually grounded speech

arXiv: https://arxiv.org/abs/2203.06937 · HTML: https://mlsanity.com/paper/2203.06937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 68 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Order-Embeddings of Images and Language](https://mlsanity.com/paper/1511.06361/) (2015)
- [Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input](https://mlsanity.com/paper/1804.01452/) (2018)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge](https://mlsanity.com/paper/2005.09409/) (2020)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Representations of language in a model of visually grounded speech signal](https://mlsanity.com/paper/1702.01991/) (2017)
- [Multimodal Convolutional Neural Networks for Matching Image and Sentence](https://mlsanity.com/paper/1504.06063/) (2015)
- [Language learning using Speech to Image retrieval](https://mlsanity.com/paper/1909.03795/) (2019)
- [Semantic speech retrieval with a visually grounded model of untranscribed speech](https://mlsanity.com/paper/1710.01949/) (2017)
- [Visually grounded cross-lingual keyword spotting in speech](https://mlsanity.com/paper/1806.05030/) (2018)
- [Visually grounded learning of keyword prediction from untranscribed speech](https://mlsanity.com/paper/1703.08136/) (2017)
- [Transfer Learning from Audio-Visual Grounding to Speech Recognition](https://mlsanity.com/paper/1907.04355/) (2019)
- [Analyzing analytical methods: The case of phonology in neural models of spoken language](https://mlsanity.com/paper/2004.07070/) (2020)
- [Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech](https://mlsanity.com/paper/1909.08491/) (2019)
- [A computational model of early language acquisition from audiovisual experiences of young infants](https://mlsanity.com/paper/1906.09832/) (2019)
- [End-to-End Multimodal Speech Recognition](https://mlsanity.com/paper/1804.09713/) (2018)
- [Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech](https://mlsanity.com/paper/2006.08387/) (2020)
- [Learning to Recognise Words using Visually Grounded Speech](https://mlsanity.com/paper/2006.00512/) (2020)
- [Fine-Grained Grounding for Multimodal Speech Recognition](https://mlsanity.com/paper/2010.02384/) (2020)
- [Semantic sentence similarity: size does not always matter](https://mlsanity.com/paper/2106.08648/) (2021)
- [Learning semantic sentence representations from visually grounded language without lexical knowledge](https://mlsanity.com/paper/1903.11393/) (2019)

## Abstract (arXiv)

Background: Computational models of speech recognition often assume that the set of target words is already given. This implies that these models do not learn to recognise speech from scratch without prior knowledge and explicit supervision. Visually grounded speech models learn to recognise speech without prior knowledge by exploiting statistical dependencies between spoken and visual input. While it has previously been shown that visually grounded speech models learn to recognise the presence of words in the input, we explicitly investigate such a model as a model of human speech recognition. Methods: We investigate the time-course of word recognition as simulated by the model using a gating paradigm to test whether its recognition is affected by well-known word-competition effects in human speech processing. We furthermore investigate whether vector quantisation, a technique for discrete representation learning, aids the model in the discovery and recognition of words. Results/Conclusion: Our experiments show that the model is able to recognise nouns in isolation and even learns to properly differentiate between plural and singular nouns. We also find that recognition is influenced by word competition from the word-initial cohort and neighbourhood density, mirroring word competition effects in human speech comprehension. Lastly, we find no evidence that vector quantisation is helpful in discovering and recognising words. Our gating experiments even show that the vector quantised model requires more of the input sequence for correct recognition.