mlsanityData to 14 Oct 2026

Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation

arXiv 2306.08920 ↗First posted 15 Jun 2023cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

The excellent generalization ability of self-supervised learning (SSL) for speech foundation models has garnered significant attention. HuBERT is a successful example that utilizes offline clustering to convert speech features into discrete units for a masked language modeling pretext task. However, simply clustering features as targets by k-means does not fully inspire the model's performance. In this work, we present an unsupervised method to improve SSL targets. Two models are proposed, MonoBERT and PolyBERT, which leverage context-independent and context-dependent phoneme-based units for pre-training. Our models outperform other SSL models significantly on the LibriSpeech benchmark without the need for iterative re-clustering and re-training. Furthermore, our models equipped with context-dependent units even outperform target-improvement models that use labeled data during pre-training. How we progressively improve the unit discovery process is demonstrated through experiments.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

31 of 38 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2022Generative Adversarial Networkscited by 5,627
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  11. 2017Improved Training of Wasserstein GANscited by 2,235
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
19 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  4. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  5. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  6. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  7. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  8. 2021Unsupervised Speech Recognitioncited by 148
  9. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  10. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  11. 2022Towards End-to-end Unsupervised Speech Recognitioncited by 45
  12. 2018Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddingscited by 22
  13. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 22
  14. 2022Speech Pre-training with Acoustic Piececited by 14
  15. 2022Supervision-Guided Codebooks for Masked Prediction in Speech Pre-trainingcited by 14
  16. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  17. 2022CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learningcited by 7
  18. 2022CTCBERT: Advancing Hidden-unit BERT with CTC Objectivescited by 5
  19. 2023Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognitioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 3; 2026: 1.

7 papers.

  1. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  2. 2023Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognitioncited by 12
  3. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  4. 2025Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0cited by 4
  5. 2025VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretrainingcited by 3
  6. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  7. 2026Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPAcited by 0