mlsanityData to 14 Oct 2026

Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning

arXiv 2510.04213 ↗First posted 5 Oct 2025eess.AS cs.SDcited by 3

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Large-scale self-supervised Pre-Trained Models (PTMs) have shown significant improvements in the speaker verification (SV) task by providing rich feature representations. In this paper, we utilize w2v-BERT 2.0, a model with approximately 600 million parameters trained on 4.5 million hours of unlabeled data across 143 languages, for the SV task. The MFA structure with Layer Adapter is employed to process the multi-layer feature outputs from the PTM and extract speaker embeddings. Additionally, we incorporate LoRA for efficient fine-tuning. Our model achieves state-of-the-art results with 0.12% and 0.55% EER on the Vox1-O and Vox1-H test sets, respectively. Furthermore, we apply knowledge distillation guided structured pruning, reducing the model size by 80% while achieving only a 0.04% EER degradation. Source code and models are released at https://github.com/ZXHY-82/w2v-BERT-2.0_SV.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 29 of its 35 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

29 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Models23 Jun
  2. 2023Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation6 Sep

Other references · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  6. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  7. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  8. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  10. 2019Structured Pruning of Large Language Modelscited by 203
  11. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  12. 2018Attentive Statistics Pooling for Deep Speaker Embeddingcited by 114
15 more
  1. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 102
  2. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 97
  3. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  4. 2019CN-CELEB: a challenging Chinese speaker recognition datasetcited by 48
  5. 2022MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verificationcited by 47
  6. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 45
  7. 2024Reshape Dimensions Network for Speaker Recognitioncited by 28
  8. 2020CN-Celeb: multi-genre speaker recognitioncited by 27
  9. 2024VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmarkcited by 26
  10. 2020The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verificationcited by 23
  11. 20243D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarizationcited by 16
  12. 2020The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020cited by 10
  13. 2024CA-MHFA: A Context-Aware Multi-Head Factorized Attentive Pooling for SSL-Based Speaker Verificationcited by 5
  14. 2024Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Modelscited by 3
  15. 2023The DKU-MSXF Speaker Verification System for the VoxCeleb Speaker Recognition Challenge 2023cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026What and When to Learn: CURriculum Ranking Loss for Large-Scale Speaker Verificationcited by 1
  2. 2026CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Modelscited by 0
  3. 2026MECT: Mixture of Experts with CNN-Transformer Network for Speaker verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  2. 2023Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification
  3. 2023Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
  4. 2024Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision
  5. 2023Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification
  6. 2025Hybrid Pruning: In-Situ Compression of Self-Supervised Speech Models for Speaker Verification and Anti-Spoofing
  7. 2026Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
  8. 2021Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems