mlsanityData to 14 Oct 2026

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

arXiv 1911.09602 ↗First posted 21 Nov 2019cs.CL cs.LG cs.SDcited by 44

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 17; 2022: 13; 2023: 6; 2024: 7; 2025: 1.

44 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  3. 2021Unsupervised Speech Recognitioncited by 148
  4. 2022Audio Self-supervised Learning: A Surveycited by 65
  5. 2021Fully Convolutional Scene Graph Generationcited by 57
  6. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  7. 2021Cross-Modal Discrete Representation Learningcited by 35
  8. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  9. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  10. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  11. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  12. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
32 more
  1. 2021CLSRIL-23: Cross Lingual Speech Representations for Indic Languagescited by 22
  2. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  3. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  4. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
  5. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  6. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  7. 2022Wav2Vec-Aug: Improved self-supervised training with limited datacited by 9
  8. 2021Cascaded Multilingual Audio-Visual Learning from Videoscited by 8
  9. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  10. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  11. 2022YFACC: A Yor`ub'a speech-image dataset for cross-lingual keyword localisation through visual groundingcited by 8
  12. 2021Double Articulation Analyzer with Prosody for Unsupervised Word and Phoneme Discoverycited by 7
  13. 2022Keyword localisation in untranscribed speech using visually grounded speech modelscited by 7
  14. 2021ZR-2021VG: Zero-Resource Speech Challenge, Visually-Grounded Language Modelling track, 2021 editioncited by 6
  15. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  16. 2021Semantic sentence similarity: size does not always mattercited by 5
  17. 2022Video-Guided Curriculum Learning for Spoken Video Groundingcited by 5
  18. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  19. 2021Discrete representations in neural models of spoken languagecited by 3
  20. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  21. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  22. 2021Spoken ObjectNet: A Bias-Controlled Spoken Caption Datasetcited by 2
  23. 2023Visually Grounded Keyword Detection and Localisation for Low-Resource Languagescited by 2
  24. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translationcited by 2
  25. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
  26. 2024Translating speech with just imagescited by 2
  27. 2024Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Unitscited by 1
  28. 2025Factorized RVQ-GAN For Disentangled Speech Tokenizationcited by 1
  29. 2022Modelling word learning and recognition using visually grounded speechcited by 0
  30. 2024Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representationscited by 0
  31. 2024Multimodal Input Aids a Bayesian Model of Phonetic Learningcited by 0
  32. 2024Visually Grounded Speech Models for Low-resource Languages and Cognitive Modellingcited by 0