mlsanityData to 14 Oct 2026

Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques

arXiv 2104.13225 ↗First posted 27 Apr 2021cs.AI cs.CL cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

This survey provides an overview of the evolution of visually grounded models of spoken language over the last 20 years. Such models are inspired by the observation that when children pick up a language, they rely on a wide range of indirect and noisy clues, crucially including signals from the visual modality co-occurring with spoken utterances. Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and Cognitive Science. The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas. We discuss the central research questions addressed, the timeline of developments, and the datasets which enabled much of this work. We then summarize the main modeling architectures and offer an exhaustive overview of the evaluation metrics and analysis techniques.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 54 of its 93 references that are in the map. First posted on arXiv 27 Apr 2021.

Builds on

54 of 93 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
38 more
  1. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  2. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  3. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  4. 2017Objects that Soundcited by 256
  5. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  6. 2019Connecting Vision and Language with Localized Narrativescited by 226
  7. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
  8. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  9. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  10. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  11. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  12. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  13. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  14. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  15. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  16. 2016Recurrent Highway Networkscited by 37
  17. 2017Representations of language in a model of visually grounded speech signalcited by 37
  18. 2016Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measurescited by 36
  19. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  20. 2017Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systemscited by 32
  21. 2017Learning Word-Like Units from Joint Audio-Visual Analysiscited by 27
  22. 2017Encoding of phonology in a recurrent neural model of grounded speechcited by 27
  23. 2018Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speechcited by 26
  24. 2019Language learning using Speech to Image retrievalcited by 21
  25. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  26. 2018Visually grounded cross-lingual keyword spotting in speechcited by 19
  27. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  28. 2017Visually grounded learning of keyword prediction from untranscribed speechcited by 16
  29. 2019Analyzing Phonetic and Graphemic Representations in End-to-End Automatic Speech Recognitioncited by 16
  30. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  31. 2020Analyzing analytical methods: The case of phonology in neural models of spoken languagecited by 16
  32. 2019Towards Visually Grounded Sub-Word Speech Unit Discoverycited by 15
  33. 2019Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanesecited by 13
  34. 2019Word Recognition, Competition, and Activation in a Model of Visually Grounded Speechcited by 13
  35. 2019Semantic query-by-example speech search using visual groundingcited by 11
  36. 2018Symbolic inductive bias for visually grounded learning of spoken languagecited by 10
  37. 2020Textual Supervision for Visually Grounded Spoken Language Understandingcited by 9
  38. 2016From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learningcited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 6; 2024: 7; 2025: 3; 2026: 2.

32 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2024Recent Advances in Speech Language Models: A Surveycited by 87
  3. 2022MAViL: Masked Audio-Video Learnerscited by 59
  4. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  5. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  6. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  7. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  8. 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
  9. 2024WaterVG: Waterway Visual Grounding based on Text-Guided Vision and mmWave Radarcited by 13
  10. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  11. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  12. 2022Learning English with Peppa Pigcited by 8
20 more
  1. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  2. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  3. 2021ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 editioncited by 6
  4. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  5. 2022Video-Guided Curriculum Learning for Spoken Video Groundingcited by 5
  6. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  7. 2025On the robustness of modeling grounded word learning through a child's egocentric inputcited by 4
  8. 2021Discrete representations in neural models of spoken languagecited by 3
  9. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  10. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  11. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  12. 2024Translating speech with just imagescited by 2
  13. 2023Visually grounded few-shot word acquisition with fewer shotscited by 1
  14. 2023UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Modelscited by 1
  15. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  16. 2025Explainable embeddings with Distance Explainercited by 1
  17. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  18. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  19. 2024Improved Visually Prompted Keyword Localisation in Real Low-Resource Settingscited by 0
  20. 2026Mapping Written Words to Spoken Words in a Different Language Using Only Visual Groundingcited by 0