mlsanityData to 7 Oct 2026

Visually Grounded Speech Models have a Mutual Exclusivity Bias

arXiv 2403.13922 ↗First posted 20 Mar 2024cs.CL eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

When children learn new words, they employ constraints such as the mutual exclusivity (ME) bias: a novel word is mapped to a novel object rather than a familiar one. This bias has been studied computationally, but only in models that use discrete word representations as input, ignoring the high variability of spoken words. We investigate the ME bias in the context of visually grounded speech models that learn from natural images and continuous speech audio. Concretely, we train a model on familiar words and test its ME bias by asking it to select between a novel and a familiar object when queried with a novel word. To simulate prior acoustic and visual knowledge, we experiment with several initialisation strategies using pretrained speech and vision networks. Our findings reveal the ME bias across the different initialisation approaches, with a stronger bias in models with more prior (in particular, visual) knowledge. Additional tests confirm the robustness of our results, even when different loss functions are considered.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 62 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

31 of 62 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  9. 2018End-to-end Audiovisual Speech Recognitioncited by 71
  10. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  11. 2020CompRess: Self-Supervised Learning by Compressing Representationscited by 61
  12. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
19 more
  1. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  2. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  3. 2017Representations of language in a model of visually grounded speech signalcited by 37
  4. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  5. 2016Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencodercited by 34
  6. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  7. 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
  8. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  9. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  10. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  11. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  12. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  13. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  14. 2018Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detectioncited by 17
  15. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  16. 2022M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrievalcited by 9
  17. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  18. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  19. 2023Visually grounded few-shot word learning in low-resource settingscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025The mutual exclusivity bias of bilingual visually grounded speech modelscited by 1
  2. 2024Visually Grounded Speech Models for Low-resource Languages and Cognitive Modellingcited by 0
  3. 2026Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Modelscited by 0