mlsanityData to 7 Oct 2026

Towards Visually Grounded Sub-Word Speech Unit Discovery

arXiv 1902.08213 ↗First posted 21 Feb 2019cs.CL cs.LG cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the manner in which interpretable sub-word speech units emerge within a convolutional neural network model trained to associate raw speech waveforms with semantically related natural image scenes. We show how diphone boundaries can be superficially extracted from the activation patterns of intermediate layers of the model, suggesting that the model may be leveraging these events for the purpose of word recognition. We present a series of experiments investigating the information encoded by these events.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 4; 2023: 0; 2024: 1.

15 papers.

Sort:

  1. 2021Unsupervised Speech Recognitioncited by 147
  2. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  3. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  4. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  5. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  6. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  7. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  8. 2022Learning English with Peppa Pigcited by 8
  9. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  10. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  11. 2021Interpreting intermediate convolutional layers of generative CNNs trained on waveformscited by 6
  12. 2021Unsupervised Automatic Speech Recognition: A Reviewcited by 5
3 more
  1. 2021Interpreting intermediate convolutional layers in unsupervised acoustic word classificationcited by 3
  2. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  3. 2021Improving Word Recognition in Speech Transcriptions by Decision-level Fusion of Stemming and Two-way Phoneme Pruningcited by 0