mlsanityData to 14 Oct 2026

Modelling word learning and recognition using visually grounded speech

arXiv 2203.06937 ↗First posted 14 Mar 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Background: Computational models of speech recognition often assume that the set of target words is already given. This implies that these models do not learn to recognise speech from scratch without prior knowledge and explicit supervision. Visually grounded speech models learn to recognise speech without prior knowledge by exploiting statistical dependencies between spoken and visual input. While it has previously been shown that visually grounded speech models learn to recognise the presence of words in the input, we explicitly investigate such a model as a model of human speech recognition. Methods: We investigate the time-course of word recognition as simulated by the model using a gating paradigm to test whether its recognition is affected by well-known word-competition effects in human speech processing. We furthermore investigate whether vector quantisation, a technique for discrete representation learning, aids the model in the discovery and recognition of words. Results/Conclusion: Our experiments show that the model is able to recognise nouns in isolation and even learns to properly differentiate between plural and singular nouns. We also find that recognition is influenced by word competition from the word-initial cohort and neighbourhood density, mirroring word competition effects in human speech comprehension. Lastly, we find no evidence that vector quantisation is helpful in discovering and recognising words. Our gating experiments even show that the vector quantised model requires more of the input sequence for correct recognition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 68 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

32 of 68 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  7. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  8. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  9. 2015Order-Embeddings of Images and Languagecited by 113
  10. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  11. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  12. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
20 more
  1. 2017Predicting Visual Features from Text for Image and Video Caption Retrievalcited by 53
  2. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  3. 2017Representations of language in a model of visually grounded speech signalcited by 37
  4. 2015Multimodal Convolutional Neural Networks for Matching Image and Sentencecited by 25
  5. 2019Language learning using Speech to Image retrievalcited by 21
  6. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  7. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  8. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  9. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  10. 2020Analyzing analytical methods: The case of phonology in neural models of spoken languagecited by 16
  11. 2019Word Recognition, Competition, and Activation in a Model of Visually Grounded Speechcited by 13
  12. 2019A computational model of early language acquisition from audiovisual experiences of young infantscited by 10
  13. 2018End-to-End Multimodal Speech Recognitioncited by 9
  14. 2020Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speechcited by 9
  15. 2020Learning to Recognise Words using Visually Grounded Speechcited by 7
  16. 2020Fine-Grained Grounding for Multimodal Speech Recognitioncited by 5
  17. 2021Semantic sentence similarity: size does not always mattercited by 5
  18. 2019Learning semantic sentence representations from visually grounded language without lexical knowledgecited by 4
  19. 2017Order embeddings and character-level convolutions for multimodal alignmentcited by 2
  20. 2018On the difficulty of a distributional semantics of spoken languagecited by 1

Built on by

Not cited yet.