# Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruning

arXiv: https://arxiv.org/abs/2510.04213 · HTML: https://mlsanity.com/paper/2510.04213/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 29 of 29)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Structured Pruning of Large Language Models](https://mlsanity.com/paper/1910.04732/) (2019)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [CN-CELEB: a challenging Chinese speaker recognition dataset](https://mlsanity.com/paper/1911.01799/) (2019)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training](https://mlsanity.com/paper/2110.05752/) (2021)
- [Reshape Dimensions Network for Speaker Recognition](https://mlsanity.com/paper/2407.18223/) (2024)
- [CN-Celeb: multi-genre speaker recognition](https://mlsanity.com/paper/2012.12468/) (2020)
- [VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark](https://mlsanity.com/paper/2407.11510/) (2024)
- [The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification](https://mlsanity.com/paper/2010.11255/) (2020)
- [3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization](https://mlsanity.com/paper/2403.19971/) (2024)
- [The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.12731/) (2020)
- [Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Models](https://mlsanity.com/paper/2506.18623/) (2025)
- [Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation](https://mlsanity.com/paper/2309.03019/) (2023)
- [CA-MHFA: A Context-Aware Multi-Head Factorized Attentive Pooling for SSL-Based Speaker Verification](https://mlsanity.com/paper/2409.15234/) (2024)
- [Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models](https://mlsanity.com/paper/2408.15585/) (2024)
- [The DKU-MSXF Speaker Verification System for the VoxCeleb Speaker Recognition Challenge 2023](https://mlsanity.com/paper/2308.08766/) (2023)

## Built on by (most cited first, 3 of 3)

- [What and When to Learn: CURriculum Ranking Loss for Large-Scale Speaker Verification](https://mlsanity.com/paper/2603.24432/) (2026)
- [CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Models](https://mlsanity.com/paper/2609.14956/) (2026)
- [MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification](https://mlsanity.com/paper/2609.24061/) (2026)

## Abstract (arXiv)

Large-scale self-supervised Pre-Trained Models (PTMs) have shown significant improvements in the speaker verification (SV) task by providing rich feature representations. In this paper, we utilize w2v-BERT 2.0, a model with approximately 600 million parameters trained on 4.5 million hours of unlabeled data across 143 languages, for the SV task. The MFA structure with Layer Adapter is employed to process the multi-layer feature outputs from the PTM and extract speaker embeddings. Additionally, we incorporate LoRA for efficient fine-tuning. Our model achieves state-of-the-art results with 0.12% and 0.55% EER on the Vox1-O and Vox1-H test sets, respectively. Furthermore, we apply knowledge distillation guided structured pruning, reducing the model size by 80% while achieving only a 0.04% EER degradation. Source code and models are released at https://github.com/ZXHY-82/w2v-BERT-2.0_SV.