mlsanityData to 7 Oct 2026

Visually Grounded Speech Models for Low-resource Languages and Cognitive Modelling

arXiv 2409.02865 ↗First posted 3 Sep 2024cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This dissertation examines visually grounded speech (VGS) models that learn from unlabelled speech paired with images. It focuses on applications for low-resource languages and understanding human language acquisition. We introduce a task called visually prompted keyword localisation to detect and localise keywords in speech using images. We demonstrate the effectiveness of VGS models in few-shot learning scenarios for low-resource languages like Yoruba. Additionally, we examine the mutual exclusivity bias in VGS models. Our monolingual VGS model exhibits this bias, but we found that multilingualism does not affect the bias in this VGS model similarly to what is observed in children.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 144 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

17 of 144 references.

References · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  8. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  9. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  10. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  11. 2017A Simple Neural Attentive Meta-Learnercited by 22
  12. 2020Few-Shot Keyword Spotting With Prototypical Networkscited by 18
5 more
  1. 2017Attentive Recurrent Comparatorscited by 13
  2. 2020Towards localisation of keywords in speech using weak supervisioncited by 7
  3. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  4. 2023Visually Grounded Keyword Detection and Localisation for Low-Resource Languagescited by 2
  5. 2020Competition in Cross-situational Word Learning: A Computational Studycited by 1

Built on by

Not cited yet.