# DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning

arXiv: https://arxiv.org/abs/2305.10005 · HTML: https://mlsanity.com/paper/2305.10005/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 39 of its 47 references that are in the map.
- First posted on arXiv 17 May 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization](https://mlsanity.com/paper/2012.06659/) (2020)
- [SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities](https://mlsanity.com/paper/2203.06849/) (2022)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)

## Built on by (most cited first, 30 of 37)

- [Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models](https://mlsanity.com/paper/2411.04996/) (2024)
- [Towards Robust Speech Representation Learning for Thousands of Languages](https://mlsanity.com/paper/2407.00837/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [SSR: Alignment-Aware Modality Connector for Speech Language Models](https://mlsanity.com/paper/2410.00168/) (2024)
- [SSDM: Scalable Speech Dysfluency Modeling](https://mlsanity.com/paper/2408.16221/) (2024)
- [Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning](https://mlsanity.com/paper/2512.19687/) (2025)
- [Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Play](https://mlsanity.com/paper/2505.02707/) (2025)
- [DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models](https://mlsanity.com/paper/2410.24177/) (2024)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces](https://mlsanity.com/paper/2311.09117/) (2023)
- [Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity](https://mlsanity.com/paper/2501.16295/) (2025)
- [USAD: Universal Speech and Audio Representation via Distillation](https://mlsanity.com/paper/2506.18843/) (2025)
- [Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective](https://mlsanity.com/paper/2401.08833/) (2024)
- [The Effect of Batch Size on Contrastive Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2402.13723/) (2024)
- [DM-Codec: Distilling Multimodal Representations for Speech Tokenization](https://mlsanity.com/paper/2410.15017/) (2024)
- [UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation](https://mlsanity.com/paper/2503.00733/) (2025)
- [Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation](https://mlsanity.com/paper/2504.18539/) (2025)
- [Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis](https://mlsanity.com/paper/2505.21138/) (2025)
- [AV-Dialog: Spoken Dialogue Models with Audio-Visual Input](https://mlsanity.com/paper/2511.11124/) (2025)
- [Sustainable self-supervised learning for speech representations](https://mlsanity.com/paper/2406.07696/) (2024)
- [EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2410.13179/) (2024)
- [StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs](https://mlsanity.com/paper/2509.22220/) (2025)
- [K*-Means: A Parameter-free Clustering Algorithm](https://mlsanity.com/paper/2505.11904/) (2025)
- [Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs](https://mlsanity.com/paper/2511.16639/) (2025)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learning](https://mlsanity.com/paper/2401.01473/) (2024)
- [Advanced Clustering Techniques for Speech Signal Enhancement: A Review and Metanalysis of Fuzzy C-Means, K-Means, and Kernel Fuzzy C-Means Methods](https://mlsanity.com/paper/2409.19448/) (2024)
- [Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers](https://mlsanity.com/paper/2506.21712/) (2025)
- [Domain Knowledge is Power: Leveraging Physiological Priors for Self Supervised Representation Learning in Electrocardiography](https://mlsanity.com/paper/2509.08116/) (2025)
- [MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery](https://mlsanity.com/paper/2512.19612/) (2025)

## Abstract (arXiv)

In this paper, we introduce self-distillation and online clustering for self-supervised speech representation learning (DinoSR) which combines masked language modeling, self-distillation, and online clustering. We show that these concepts complement each other and result in a strong representation learning model for speech. DinoSR first extracts contextualized embeddings from the input audio with a teacher network, then runs an online clustering system on the embeddings to yield a machine-discovered phone inventory, and finally uses the discretized tokens to guide a student network. We show that DinoSR surpasses previous state-of-the-art performance in several downstream tasks, and provide a detailed analysis of the model and the learned discrete units.