mlsanityData to 14 Oct 2026

Pushing the Frontiers of Self-Distillation Prototypes Network with Dimension Regularization and Score Normalization

arXiv 2505.13826 ↗First posted 20 May 2025eess.AS cs.SDcited by 1

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Developing robust speaker verification (SV) systems without speaker labels has been a longstanding challenge. Earlier research has highlighted a considerable performance gap between self-supervised and fully supervised approaches. In this paper, we enhance the non-contrastive self-supervised framework, Self-Distillation Prototypes Network (SDPN), by introducing dimension regularization that explicitly addresses the collapse problem through the application of regularization terms to speaker embeddings. Moreover, we integrate score normalization techniques from fully supervised SV to further bridge the gap toward supervised verification performance. SDPN with dimension regularization and score normalization sets a new state-of-the-art on the VoxCeleb1 speaker verification evaluation benchmark, achieving Equal Error Rate 1.29%, 1.60%, and 2.80% for trial VoxCeleb1-{O,E,H} respectively. These results demonstrate relative improvements of 28.3%, 19.6%, and 22.6% over the current best self-supervised methods, thereby advancing the frontiers of SV technology.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 16 of its 24 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

16 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision17 Jun

Other references · 15

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  8. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 108
  9. 2020Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learningcited by 33
  10. 2021Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularizationcited by 17
  11. 2022Pushing the limits of self-supervised speaker verification using regularized distillation frameworkcited by 15
  12. 2023Disentangling Voice and Content with Self-Supervision for Speaker Recognitioncited by 13
3 more
  1. 2022C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verificationcited by 12
  2. 2022Contrastive-mixup learning for improved speaker verificationcited by 7
  3. 2022Curriculum learning for self-supervised speaker verificationcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Interpolating Speaker Identities in Embedding Space for Data Expansioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
  2. 2023Self-Supervised Learning with Cluster-Aware-DINO for High-Performance Robust Speaker Verification
  3. 2023Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification
  4. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  5. 2021Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems
  6. 2025Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning
  7. 2022Label-free Knowledge Distillation with Contrastive Loss for Light-weight Speaker Recognition
  8. 2026Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach