mlsanityData to 14 Oct 2026

Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge

arXiv 2210.15759 ↗First posted 27 Oct 2022cs.CL cs.SD eess.AScited by 34

Not on the map. Why?

Abstract (arXiv)

Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks -- Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling -- and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 63 of its 140 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

63 of 140 references.

References · 63

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2017Look, Listen and Learncited by 458
  9. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  10. 2022Masked Autoencoders that Listencited by 273
  11. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  12. 2021Direct speech-to-speech translation with discrete unitscited by 149
38 more
  1. 2022Generative Spoken Dialogue Language Modelingcited by 126
  2. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  3. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  4. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  5. 2020Unsupervised pretraining transfers well across languagescited by 101
  6. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  7. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  8. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  9. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  10. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  11. 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
  12. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  13. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  14. 2018Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speechcited by 52
  15. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  16. 2017The Zero Resource Speech Challenge 2017cited by 45
  17. 2016SimVerb-3500: A Large-Scale Evaluation Set of Verb Similaritycited by 44
  18. 2020The Zero Resource Speech Challenge 2020: Discovering discrete subword and word unitscited by 39
  19. 2019The Zero Resource Speech Challenge 2019: TTS without Tcited by 38
  20. 2017Representations of language in a model of visually grounded speech signalcited by 37
  21. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 33
  22. 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
  23. 2017An embedded segmental K-means model for unsupervised segmentation and clustering of speechcited by 31
  24. 2022Do self-supervised speech models develop human-like perception biases?cited by 28
  25. 2016A segmental framework for fully-unsupervised large-vocabulary speech recognitioncited by 27
  26. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  27. 2021Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processingcited by 26
  28. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  29. 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
  30. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  31. 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
  32. 2020Evaluating the reliability of acoustic speech embeddingscited by 21
  33. 2019Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networkscited by 20
  34. 2021Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and wordscited by 18
  35. 2020Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metricscited by 16
  36. 2021Unsupervised Speech Segmentation and Variable Rate Representation Learning using Segmental Contrastive Predictive Codingcited by 14
  37. 2019Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversioncited by 12
  38. 2020Self-Expressing Autoencoders for Unsupervised Spoken Term Discoverycited by 12

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 15; 2025: 5; 2026: 6.

34 papers.

Sort:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2023Voice Conversion With Just Nearest Neighborscited by 68
  3. 2023Generative Spoken Language Model based on continuous word-sized audio tokenscited by 19
  4. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  5. 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
  6. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  7. 2025Emergent morpho-phonological representations in self-supervised speech modelscited by 11
  8. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  10. 2025fastabx: A library for efficient computation of ABX discriminabilitycited by 7
  11. 2024Zero-resource Speech Translation and Recognition with LLMscited by 6
  12. 2024Anonymising Elderly and Pathological Speech: Voice Conversion Using DDSP and Query-by-Examplecited by 5
22 more
  1. 2026ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modelingcited by 5
  2. 2026An Empirical Recipe for Universal Phone Recognitioncited by 5
  3. 2024STAB: Speech Tokenizer Assessment Benchmarkcited by 4
  4. 2023A benchmark for computational analysis of animal behavior, using animal-borne tagscited by 3
  5. 2024Investigating the 'Autoencoder Behavior' in Speech Self-Supervised Models: a focus on HuBERT's Pretrainingcited by 3
  6. 2024A model of early word acquisition based on realistic-scale audiovisual naming eventscited by 3
  7. 2024kNN Retrieval for Simple and Effective Zero-Shot Multi-speaker Text-to-Speechcited by 3
  8. 2026Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representationscited by 3
  9. 2023Simultaneous or Sequential Training? How Speech Representations Cooperate in a Multi-Task Self-Supervised Learning Systemcited by 2
  10. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
  11. 2023Bigger is not Always Better: The Effect of Context Size on Speech Pre-Trainingcited by 2
  12. 2023XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into wordscited by 1
  13. 2024QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversioncited by 1
  14. 2024Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Modelcited by 1
  15. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  16. 2026Computational modeling of early language learning from acoustic speech and audiovisual input without linguistic priorscited by 1
  17. 2024Age-Dependent Analysis and Stochastic Generation of Child-Directed Speechcited by 0
  18. 2024Simulating Articulatory Trajectories with Phonological Feature Interpolationcited by 0
  19. 2024Textless NLP -- Zero Resource Challenge with Low Resource Computecited by 0
  20. 2024Developmental Predictive Coding Model for Early Infancy Mono and Bilingual Vocal Continual Learningcited by 0
  21. 2026DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Unitscited by 0
  22. 2026Language Discrimination Improves Linguistic Learning in Multilingual Speech Modelscited by 0