# Visually Grounded Speech Models for Low-resource Languages and Cognitive Modelling

arXiv: https://arxiv.org/abs/2409.02865 · HTML: https://mlsanity.com/paper/2409.02865/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 144 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech](https://mlsanity.com/paper/1911.09602/) (2019)
- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [A Simple Neural Attentive Meta-Learner](https://mlsanity.com/paper/1707.03141/) (2017)
- [Few-Shot Keyword Spotting With Prototypical Networks](https://mlsanity.com/paper/2007.14463/) (2020)
- [Attentive Recurrent Comparators](https://mlsanity.com/paper/1703.00767/) (2017)
- [Towards localisation of keywords in speech using weak supervision](https://mlsanity.com/paper/2012.07396/) (2020)
- [Visually Grounded Speech Models have a Mutual Exclusivity Bias](https://mlsanity.com/paper/2403.13922/) (2024)
- [Visually Grounded Keyword Detection and Localisation for Low-Resource Languages](https://mlsanity.com/paper/2302.00765/) (2023)
- [Competition in Cross-situational Word Learning: A Computational Study](https://mlsanity.com/paper/2012.03370/) (2020)

## Abstract (arXiv)

This dissertation examines visually grounded speech (VGS) models that learn from unlabelled speech paired with images. It focuses on applications for low-resource languages and understanding human language acquisition. We introduce a task called visually prompted keyword localisation to detect and localise keywords in speech using images. We demonstrate the effectiveness of VGS models in few-shot learning scenarios for low-resource languages like Yoruba. Additionally, we examine the mutual exclusivity bias in VGS models. Our monolingual VGS model exhibits this bias, but we found that multilingualism does not affect the bias in this VGS model similarly to what is observed in children.