# An attention-based backend allowing efficient fine-tuning of transformer models for speaker verification

arXiv: https://arxiv.org/abs/2210.01273 · HTML: https://mlsanity.com/paper/2210.01273/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2022.

## Built on by (most cited first, 16 of 16)

- [Exploring WavLM Back-ends for Speech Spoofing and Deepfake Detection](https://mlsanity.com/paper/2409.05032/) (2024)
- [Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation](https://mlsanity.com/paper/2309.03019/) (2023)
- [A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/2407.04966/) (2024)
- [Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models](https://mlsanity.com/paper/2406.02285/) (2024)
- [TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2505.06660/) (2025)
- [Evolutionary Multi-Objective Fusion of Deepfake Speech Detectors](https://mlsanity.com/paper/2604.01330/) (2026)
- [Improving Out-of-Domain Audio Deepfake Detection via Layer Selection and Fusion of SSL-Based Countermeasures](https://mlsanity.com/paper/2509.12003/) (2025)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification](https://mlsanity.com/paper/2512.22148/) (2025)
- [Enhancing Multi-Corpus Training in SSL-Based Anti-Spoofing Models: Domain-Invariant Feature Extraction](https://mlsanity.com/paper/2603.18657/) (2026)
- [VARAN: Variational Inference for Self-Supervised Speech Models Fine-Tuning on Downstream Tasks](https://mlsanity.com/paper/2508.12061/) (2025)
- [Self-Supervised Learning for Speaker Recognition: A study and review](https://mlsanity.com/paper/2602.10829/) (2026)
- [SpAArSIST: Sparsified AASIST for Efficient and Reliable Anti-Spoofing](https://mlsanity.com/paper/2606.11674/) (2026)
- [From Self-Supervised Speech Models to Mixture-of-Experts for Robust Anti-Spoofing](https://mlsanity.com/paper/2606.14639/) (2026)
- [Kiwano: A Cutting-Edge Open-Source Toolkit for Speaker Verification](https://mlsanity.com/paper/2606.22369/) (2026)
- [Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors](https://mlsanity.com/paper/2610.16826/) (2026)

## Abstract (arXiv)

In recent years, self-supervised learning paradigm has received extensive attention due to its great success in various down-stream tasks. However, the fine-tuning strategies for adapting those pre-trained models to speaker verification task have yet to be fully explored. In this paper, we analyze several feature extraction approaches built on top of a pre-trained model, as well as regularization and learning rate schedule to stabilize the fine-tuning process and further boost performance: multi-head factorized attentive pooling is proposed to factorize the comparison of speaker representations into multiple phonetic clusters. We regularize towards the parameters of the pre-trained model and we set different learning rates for each layer of the pre-trained model during fine-tuning. The experimental results show our method can significantly shorten the training time to 4 hours and achieve SOTA performance: 0.59%, 0.79% and 1.77% EER on Vox1-O, Vox1-E and Vox1-H, respectively.