mlsanityData to 14 Oct 2026

Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model

arXiv 2305.11435 ↗First posted 19 May 2023eess.AS cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we show that representations capturing syllabic units emerge when training a self-supervised speech model with a visually-grounded training objective. We demonstrate that a nearly identical model architecture (HuBERT) trained with a masked language modeling loss does not exhibit this same ability, suggesting that the visual grounding objective is responsible for the emergence of this phenomenon. We propose the use of a minimum cut algorithm to automatically predict syllable boundaries in speech, followed by a 2-stage clustering method to group identical syllables together. We show that our model not only outperforms a state-of-the-art syllabic segmentation method on the language it was trained on (English), but also generalizes in a zero-shot fashion to Estonian. Finally, we show that the same model is capable of zero-shot generalization for a word segmentation task on 4 other languages from the Zerospeech Challenge, in some cases beating the previous state-of-the-art.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

42 of 55 references.

References · 42

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  8. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  9. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  10. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  11. 2022Generative Spoken Dialogue Language Modelingcited by 126
  12. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
30 more
  1. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  2. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 51
  3. 2019Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speechcited by 44
  4. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  5. 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
  6. 2021Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniquescited by 32
  7. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  8. 2017An embedded segmental K-means model for unsupervised segmentation and clustering of speechcited by 31
  9. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  10. 2017Encoding of phonology in a recurrent neural model of grounded speechcited by 27
  11. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  12. 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
  13. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  14. 2017Semantic speech retrieval with a visually grounded model of untranscribed speechcited by 20
  15. 2021Talk, Don't Write: A Study of Direct Speech-Based Image Retrievalcited by 19
  16. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  17. 2021Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and wordscited by 18
  18. 2018Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detectioncited by 17
  19. 2020Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metricscited by 16
  20. 2019Models of Visually Grounded Speech Signal Pay Attention To Nouns: a Bilingual Experiment on English and Japanesecited by 13
  21. 2019Word Recognition, Competition, and Activation in a Model of Visually Grounded Speechcited by 13
  22. 2020Self-Expressing Autoencoders for Unsupervised Spoken Term Discoverycited by 12
  23. 2021Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigationcited by 12
  24. 2022DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexiconcited by 12
  25. 2019Semantic query-by-example speech search using visual groundingcited by 11
  26. 2019A computational model of early language acquisition from audiovisual experiences of young infantscited by 10
  27. 2021Evaluation of Audio-Visual Alignments in Visually Grounded Speech Modelscited by 8
  28. 2022Learning English with Peppa Pigcited by 8
  29. 2015Nonparametric Bayesian Double Articulation Analyzer for Direct Language Acquisition from Continuous Speech Signalscited by 6
  30. 2023Visually Grounded Keyword Detection and Localisation for Low-Resource Languagescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 1; 2026: 4.

11 papers.

  1. 2024Sylber: Syllabic Embedding Representation of Speech from Raw Audiocited by 19
  2. 2023Visually grounded few-shot word learning in low-resource settingscited by 4
  3. 2024Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERTcited by 4
  4. 2025Should Top-Down Clustering Affect Boundaries in Unsupervised Word Discovery?cited by 4
  5. 2024Visually Grounded Speech Models have a Mutual Exclusivity Biascited by 3
  6. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  7. 2023XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into wordscited by 1
  8. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  9. 2026findsylls: A Language-Agnostic Toolkit for Syllable-Level Speech Tokenization and Embeddingcited by 0
  10. 2026Recovering the Zipfian Distribution in Unsupervised Term Discoverycited by 0
  11. 2026Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenizationcited by 0