# Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision

arXiv: https://arxiv.org/abs/2308.02774 · HTML: https://mlsanity.com/paper/2308.02774/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 5 Aug 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning](https://mlsanity.com/paper/2012.07178/) (2020)
- [Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularization](https://mlsanity.com/paper/2112.04459/) (2021)
- [Pushing the limits of self-supervised speaker verification using regularized distillation framework](https://mlsanity.com/paper/2211.04168/) (2022)
- [Self-supervised Speaker Recognition with Loss-gated Learning](https://mlsanity.com/paper/2110.03869/) (2021)
- [C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verification](https://mlsanity.com/paper/2208.07446/) (2022)
- [Self-Supervised Learning with Cluster-Aware-DINO for High-Performance Robust Speaker Verification](https://mlsanity.com/paper/2304.05754/) (2023)
- [A comprehensive study on self-supervised distillation for speaker representation learning](https://mlsanity.com/paper/2210.15936/) (2022)
- [Contrastive-mixup learning for improved speaker verification](https://mlsanity.com/paper/2202.10672/) (2022)
- [Curriculum learning for self-supervised speaker verification](https://mlsanity.com/paper/2203.14525/) (2022)

## Abstract (arXiv)

Training speaker-discriminative and robust speaker verification systems without explicit speaker labels remains a persistent challenge. In this paper, we propose a novel self-supervised speaker verification approach, Self-Distillation Prototypes Network (SDPN), which effectively facilitates self-supervised speaker representation learning. SDPN assigns the representation of the augmented views of an utterance to the same prototypes as the representation of the original view, thereby enabling effective knowledge transfer between the augmented and original views. Due to lack of negative pairs in the SDPN training process, the network tends to align positive pairs quite closely in the embedding space, a phenomenon known as model collapse. To mitigate this problem, we introduce a diversity regularization term to embeddings in SDPN. Comprehensive experiments on the VoxCeleb datasets demonstrate the superiority of SDPN among self-supervised speaker verification approaches. SDPN sets a new state-of-the-art on the VoxCeleb1 speaker verification evaluation benchmark, achieving Equal Error Rate 1.80%, 1.99%, and 3.62% for trial VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H, without using any speaker labels in training. Ablation studies show that both proposed learnable prototypes in self-distillation network and diversity regularization contribute to the verification performance.