mlsanityData to 7 Oct 2026

YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual grounding

arXiv 2210.04600 ↗First posted 10 Oct 2022cs.CL eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

Visually grounded speech (VGS) models are trained on images paired with unlabelled spoken captions. Such models could be used to build speech systems in settings where it is impossible to get labelled data, e.g. for documenting unwritten languages. However, most VGS studies are in English or other high-resource languages. This paper attempts to address this shortcoming. We collect and release a new single-speaker dataset of audio captions for 6k Flickr images in Yor`ub'a -- a real low-resource language spoken in Nigeria. We train an attention-based VGS model where images are automatically tagged with English visual labels and paired with Yor`ub'a utterances. This enables cross-lingual keyword localisation: a written English query is detected and located in Yor`ub'a speech. To quantify the effect of the smaller dataset, we compare to English systems trained on similar and more data. We hope that this new dataset will stimulate research in the use of VGS models for real low-resource languages.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

28 of 45 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  7. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  10. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  11. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  12. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
16 more
  1. 2017Representations of language in a model of visually grounded speech signalcited by 37
  2. 2022BibleTTS: a large, high-fidelity, multilingual, and uniquely African speech corpuscited by 31
  3. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  4. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  5. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  6. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  7. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  8. 2019Towards Visually Grounded Sub-Word Speech Unit Discoverycited by 15
  9. 2022Building African Voicescited by 14
  10. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  11. 2020Direct Speech-to-image Translationcited by 12
  12. 2017Towards speech-to-text translation without speech recognitioncited by 11
  13. 2018Multimodal One-Shot Learning of Speech and Imagescited by 11
  14. 2019On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrievalcited by 8
  15. 2020Unsupervised vs. transfer learning for multimodal one-shot matching of speech and imagescited by 7
  16. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 3; 2026: 1.

8 papers.

  1. 2023AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASRcited by 34
  2. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  3. 2025The NaijaVoices Dataset: Cultivating Large-Scale, High-Quality, Culturally-Rich Speech Data for African Languagescited by 7
  4. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  5. 2024Translating speech with just imagescited by 2
  6. 2025Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospectscited by 2
  7. 2026Connecting Speech to Words through Imagescited by 1
  8. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0