mlsanityData to 14 Oct 2026

Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation

arXiv 2309.03019 ↗First posted 6 Sep 2023eess.AScited by 6

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

This paper explores the use of ASR-pretrained Conformers for speaker verification, leveraging their strengths in modeling speech signals. We introduce three strategies: (1) Transfer learning to initialize the speaker embedding network, improving generalization and reducing overfitting. (2) Knowledge distillation to train a more flexible speaker verification model, incorporating frame-level ASR loss as an auxiliary task. (3) A lightweight speaker adaptor for efficient feature conversion without altering the original ASR Conformer, allowing parallel ASR and speaker verification. Experiments on VoxCeleb show significant improvements: transfer learning yields a 0.48% EER, knowledge distillation results in a 0.43% EER, and the speaker adaptor approach, with just an added 4.92M parameters to a 130.94M-parameter model, achieves a 0.57% EER. Overall, our methods effectively transfer ASR capabilities to speaker verification tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 41 of its 70 references that are in the map. First posted on arXiv 6 Sep 2023.

Builds on

41 of 70 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017Squeeze-and-Excitation Networkscited by 4,789
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
29 more
  1. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  2. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  3. 2017Learning multiple visual domains with residual adapterscited by 557
  4. 2019Res2Net: A New Multi-scale Backbone Architecturecited by 470
  5. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  6. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  7. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  8. 2020In defence of metric learning for speaker recognitioncited by 126
  9. 2018Attentive Statistics Pooling for Deep Speaker Embeddingcited by 114
  10. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  11. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  12. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 93
  13. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  14. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  15. 2018Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition Systemcited by 47
  16. 2022MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verificationcited by 47
  17. 2020Optimizing Deeper Transformers on Small Datasetscited by 42
  18. 2021Fine-tuning wav2vec2 for speaker recognitioncited by 39
  19. 2020Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognitioncited by 35
  20. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 32
  21. 2019Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognitioncited by 27
  22. 2020The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verificationcited by 23
  23. 2022An attention-based backend allowing efficient fine-tuning of transformer models for speaker verificationcited by 16
  24. 2018Speaker Embedding Extraction with Phonetic Informationcited by 10
  25. 2020The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020cited by 10
  26. 2022Parameter-efficient transfer learning of pre-trained Transformer models for speaker verification using adapterscited by 9
  27. 2022Parameter Efficient Transfer Learning for Various Speech Processing Taskscited by 9
  28. 2022Robust Speaker Recognition with Transformers Using wav2vec 2.0cited by 5
  29. 2022Towards A Unified Conformer Structure: from ASR to ASV Taskcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  2. 2025LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Modelscited by 5
  3. 2025Enhancing Speaker Verification with w2v-BERT 2.0 and Knowledge Distillation guided Structured Pruningcited by 3
  4. 2024Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learningcited by 1
  5. 2024Automatic Speech Recognition Advancements for Indigenous Languages of the Americascited by 1
  6. 2026CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Text-Independent Speaker Verification Using Discrete Audio Tokens ↗
  2. 2023Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification
  3. 2026Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
  4. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  5. 2023Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification
  6. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  7. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  8. 2021Multi-Level Transfer Learning from Near-Field to Far-Field Speaker Verification