mlsanityData to 14 Oct 2026

Label-free Knowledge Distillation with Contrastive Loss for Light-weight Speaker Recognition

arXiv 2212.03090 ↗First posted 6 Dec 2022cs.SD eess.AScited by 1

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Very deep models for speaker recognition (SR) have demonstrated remarkable performance improvement in recent research. However, it is impractical to deploy these models for on-device applications with constrained computational resources. On the other hand, light-weight models are highly desired in practice despite their sub-optimal performance. This research aims to improve light-weight SR models through large-scale label-free knowledge distillation (KD). Existing KD approaches for SR typically require speaker labels to learn task-specific knowledge, due to the inefficiency of conventional loss for distillation. To address the inefficiency problem and achieve label-free KD, we propose to employ the contrastive loss from self-supervised learning for distillation. Extensive experiments are conducted on a collection of public speech datasets from diverse sources. Results on light-weight SR models show that the proposed approach of label-free KD with contrastive loss consistently outperforms both conventional distillation methods and self-supervised learning methods by a significant margin.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 32 of its 39 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

32 of 39 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2019Searching for MobileNetV3cited by 1,994
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  11. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  12. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
20 more
  1. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  2. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  3. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  4. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  5. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  6. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  7. 2020In defence of metric learning for speaker recognitioncited by 126
  8. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 122
  9. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  10. 2019BUT System Description to VoxCeleb Speaker Recognition Challenge 2019cited by 54
  11. 2020Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learningcited by 33
  12. 2019Large Margin Softmax Loss for Speaker Verificationcited by 29
  13. 2020CN-Celeb: multi-genre speaker recognitioncited by 27
  14. 2021Towards Lightweight Applications: Asymmetric Enroll-Verify Structure for Speaker Verificationcited by 6
  15. 2020Deep Normalization for Speaker Vectorscited by 4
  16. 2020Learning Speaker Embedding with Momentum Contrastcited by 3
  17. 2020Compact Speaker Embedding: lrx-vectorcited by 3
  18. 2022Royalflush Speaker Diarization System for ICASSP 2022 Multi-channel Multi-party Meeting Transcription Challengecited by 3
  19. 2021Binary Neural Network for Speaker Verificationcited by 2
  20. 2021Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verificationcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verificationcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  2. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  3. 2023Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
  4. 2024Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
  5. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  6. 2021Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach
  7. 2023Two-Step Knowledge Distillation for Tiny Speech Enhancement
  8. 2024Integrated Multi-Level Knowledge Distillation for Enhanced Speaker Verification