mlsanityData to 7 Oct 2026

Word Discovery in Visually Grounded, Self-Supervised Speech Models

arXiv 2203.15081 ↗First posted 28 Mar 2022eess.AS cs.AI cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

We present a method for visually-grounded spoken term discovery. After training either a HuBERT or wav2vec2.0 model to associate spoken captions with natural images, we show that powerful word segmentation and clustering capability emerges within the model's self-attention heads. Our experiments reveal that this ability is not present to nearly the same extent in the base HuBERT and wav2vec2.0 models, suggesting that the visual grounding task is a crucial component of the word discovery capability we observe. We also evaluate our method on the Buckeye word segmentation and ZeroSpeech spoken term discovery tasks, where we perform on par with or better than currently published methods on several metrics. Code and model weights are available at https://github.com/jasonppy/word-discovery.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

35 of 45 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  11. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  12. 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
23 more
  1. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  2. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  3. 2017The Zero Resource Speech Challenge 2017cited by 45
  4. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  5. 2021Cross-Modal Discrete Representation Learningcited by 35
  6. 2019Large-scale representation learning from visually grounded untranscribed speechcited by 33
  7. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  8. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  9. 2017An embedded segmental K-means model for unsupervised segmentation and clustering of speechcited by 31
  10. 2016A segmental framework for fully-unsupervised large-vocabulary speech recognitioncited by 27
  11. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  12. 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
  13. 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
  14. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  15. 2021Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and wordscited by 18
  16. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  17. 2020Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metricscited by 16
  18. 2019Word Recognition, Competition, and Activation in a Model of Visually Grounded Speechcited by 13
  19. 2021Attention-Based Keyword Localisation in Speech using Visual Groundingcited by 13
  20. 2020Self-Expressing Autoencoders for Unsupervised Spoken Term Discoverycited by 12
  21. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  22. 2022Learning English with Peppa Pigcited by 8
  23. 2020Learning to Recognise Words using Visually Grounded Speechcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 10; 2024: 13; 2025: 5; 2026: 5.

37 papers.

Sort:

By area:

  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  3. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  4. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  5. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  6. 2023Putting Natural in Natural Language Processingcited by 17
  7. 2024Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Languagecited by 17
  8. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
  9. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  10. 2026[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmeticcited by 11
  11. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  12. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
25 more
  1. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 7
  2. 2022Towards visually prompted keyword localisation for zero-resource spoken languagescited by 6
  3. 2024Unsupervised Word Discovery: Boundary Detection with Clustering vs. Dynamic Programmingcited by 6
  4. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  5. 2023Hindi as a Second Language: Improving Visually Grounded Speech with Semantically Similar Samplescited by 4
  6. 2024Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERTcited by 4
  7. 2025Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?cited by 4
  8. 2024Revisiting speech segmentation and lexicon learning with better featurescited by 3
  9. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  10. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  11. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  12. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  13. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
  14. 2024SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image datacited by 2
  15. 2025Towards Unsupervised Speech Recognition at the Syllable-Levelcited by 2
  16. 2023Investigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decodingcited by 1
  17. 2023XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into wordscited by 1
  18. 2024Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Modelcited by 1
  19. 2024Towards Unsupervised Speech Recognition Without Pronunciation Modelscited by 1
  20. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1
  21. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  22. 2026Revisiting Lexicon Evaluation in Unsupervised Word Discoverycited by 1
  23. 2024Grammar Induction from Visual, Speech and Textcited by 0
  24. 2025Unifying Model and Layer Fusion for Speech Foundation Modelscited by 0
  25. 2026findsylls: A Language-Agnostic Toolkit for Syllable-Level Speech Tokenization and Embeddingcited by 0