# Improving Speaker Verification with Self-Pretrained Transformer Models

arXiv: https://arxiv.org/abs/2305.10517 · HTML: https://mlsanity.com/paper/2305.10517/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2023.

## Built on by (most cited first, 5 of 5)

- [TS-SUPERB: A Target Speech Processing Benchmark for Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2505.06660/) (2025)
- [ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks](https://mlsanity.com/paper/2407.21066/) (2024)
- [Acoustic-based Gender Differentiation in Speech-aware Language Models](https://mlsanity.com/paper/2509.21125/) (2025)
- [Towards Understanding Self-Pretraining for Sequence Classification](https://mlsanity.com/paper/2605.21070/) (2026)
- [Whisper Speaker Identification: Leveraging Pre-Trained Multilingual Transformers for Robust Speaker Embeddings](https://mlsanity.com/paper/2503.10446/) (2025)

## Abstract (arXiv)

Recently, fine-tuning large pre-trained Transformer models using downstream datasets has received a rising interest. Despite their success, it is still challenging to disentangle the benefits of large-scale datasets and Transformer structures from the limitations of the pre-training. In this paper, we introduce a hierarchical training approach, named self-pretraining, in which Transformer models are pretrained and finetuned on the same dataset. Three pre-trained models including HuBERT, Conformer and WavLM are evaluated on four different speaker verification datasets with varying sizes. Our experiments show that these self-pretrained models achieve competitive performance on downstream speaker verification tasks with only one-third of the data compared to Librispeech pretraining, such as VoxCeleb1 and CNCeleb1. Furthermore, when pre-training only on the VoxCeleb2-dev, the Conformer model outperforms the one pre-trained on 94k hours of data using the same fine-tuning settings.