mlsanityData to 7 Oct 2026

Learning Word-Like Units from Joint Audio-Visual Analysis

arXiv 1701.07481 ↗First posted 25 Jan 2017cs.CL cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the word 'lighthouse' within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 7; 2023: 4; 2024: 3; 2025: 0; 2026: 2.

27 papers.

Sort:

  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  2. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  3. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  4. 2021Cross-Modal Discrete Representation Learningcited by 35
  5. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  6. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  7. 2022TVLT: Textless Vision-Language Transformercited by 22
  8. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  9. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  10. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  11. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  12. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
15 more
  1. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
  2. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  3. 2022Learning English with Peppa Pigcited by 8
  4. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  5. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  6. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  7. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  8. 2024RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detectioncited by 4
  9. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  10. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  11. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  12. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
  13. 2022Unsupervised Multimodal Word Discovery based on Double Articulation Analysis with Co-occurrence cuescited by 1
  14. 2026Connecting Speech to Words through Imagescited by 1
  15. 2026Mapping Written Words to Spoken Words in a Different Language Using Only Visual Groundingcited by 0