mlsanityData to 14 Oct 2026

DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning

arXiv 2305.10005 ↗First posted 17 May 2023cs.CLcited by 37

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show that these concepts complement each other and result in a strong representation learning model for speech. DinoSR first extracts contextualized embeddings from the input audio with a teacher network, then runs an online clustering system on the embeddings to yield a machine-discovered phone inventory, and finally uses the discretized tokens to guide a student network. We show that DinoSR surpasses previous state-of-the-art performance in several downstream tasks, and provide a detailed analysis of the model and the learned discrete units.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 39 of its 47 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

39 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models14 Jul

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
26 more
  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  3. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  8. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  9. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  10. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  12. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  13. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  14. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  15. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  16. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  17. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  18. 2022Audio Self-supervised Learning: A Surveycited by 65
  19. 2018The challenge of realistic music generation: modelling raw audio at scalecited by 63
  20. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  21. 2018Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modelingcited by 34
  22. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  23. 2019Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learningcited by 16
  24. 2022CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representationscited by 11
  25. 2022Autoregressive Co-Training for Learning Discrete Speech Representationscited by 8
  26. 2021Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclawcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 18; 2026: 5.

37 papers.

Sort:

By area:

  1. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  2. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
  3. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  4. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  5. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  6. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  7. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 10
  8. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 9
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 8
  10. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  11. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  12. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
25 more
  1. 2024Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspectivecited by 5
  2. 2024The Effect of Batch Size on Contrastive Self-Supervised Speech Representation Learningcited by 5
  3. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  4. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  5. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  6. 2025Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysiscited by 5
  7. 2025AV-Dialog: Spoken Dialogue Models with Audio-Visual Inputcited by 5
  8. 2024Sustainable self-supervised learning for speech representationscited by 4
  9. 2024EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learningcited by 4
  10. 2025StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMscited by 4
  11. 2025K*-Means: A Parameter-free Clustering Algorithmcited by 3
  12. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  13. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  14. 2024Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learningcited by 1
  15. 2024Advanced Clustering Techniques for Speech Signal Enhancement: A Review and Metanalysis of Fuzzy C-Means, K-Means, and Kernel Fuzzy C-Means Methodscited by 1
  16. 2025Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformerscited by 1
  17. 2025Domain Knowledge is Power: Leveraging Physiological Priors for Self Supervised Representation Learning in Electrocardiographycited by 1
  18. 2025MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discoverycited by 1
  19. 2025SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptationcited by 1
  20. 2025Learning Speech Representations with Variational Predictive Codingcited by 0
  21. 2026MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representationcited by 0
  22. 2026Cluster Assignments in Soft Targets Shape Speech Representations: Evidence from S-JEPAcited by 0
  23. 2026Automatic depth-based local center clustering via β-integrated local depth and adaptive groupingcited by 0
  24. 2026Deep Speech Representations Track Word Timing along a Low-Dimensional Circular Trajectorycited by 0
  25. 2026ONCship: A Large-Scale Underwater Acoustic Corpus for Ship Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
  2. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
  3. 2023Self-Supervised Learning with Cluster-Aware-DINO for High-Performance Robust Speaker Verification
  4. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learning
  5. 2023An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speech
  6. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning
  7. 2024Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
  8. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition