# Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning

arXiv: https://arxiv.org/abs/2012.07178 · HTML: https://mlsanity.com/paper/2012.07178/

## Key facts

- Cited by 29 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 13 Dec 2020.

## Built on by (most cited first, 29 of 29)

- [Pushing the limits of raw waveform speaker recognition](https://mlsanity.com/paper/2203.08488/) (2022)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularization](https://mlsanity.com/paper/2112.04459/) (2021)
- [Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning](https://mlsanity.com/paper/2207.05506/) (2022)
- [Non-Contrastive Self-supervised Learning for Utterance-Level Information Extraction from Speech](https://mlsanity.com/paper/2208.05445/) (2022)
- [What Do Self-Supervised Speech and Speaker Models Learn? New Findings From a Cross Model Layer-Wise Analysis](https://mlsanity.com/paper/2401.17632/) (2024)
- [Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT](https://mlsanity.com/paper/2205.07180/) (2022)
- [Singer Identity Representation Learning using Self-Supervised Techniques](https://mlsanity.com/paper/2401.05064/) (2024)
- [Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models](https://mlsanity.com/paper/2406.02285/) (2024)
- [Non-Contrastive Self-Supervised Learning of Utterance-Level Speech Representations](https://mlsanity.com/paper/2208.05413/) (2022)
- [Cross-Corpora Spoken Language Identification with Domain Diversification and Generalization](https://mlsanity.com/paper/2302.05110/) (2023)
- [Experimenting with Additive Margins for Contrastive Self-Supervised Speaker Verification](https://mlsanity.com/paper/2306.03664/) (2023)
- [Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker Representations](https://mlsanity.com/paper/2404.14913/) (2024)
- [Self-Supervised Frameworks for Speaker Verification via Bootstrapped Positive Sampling](https://mlsanity.com/paper/2501.17772/) (2025)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [The JHU submission to VoxSRC-21: Track 3](https://mlsanity.com/paper/2109.13425/) (2021)
- [NASTAR: Noise Adaptive Speech Enhancement with Target-Conditional Resampling](https://mlsanity.com/paper/2206.09058/) (2022)
- [Multi-source Domain Adaptation for Text-independent Forensic Speaker Recognition](https://mlsanity.com/paper/2211.09913/) (2022)
- [SSPS: Self-Supervised Positive Sampling for Robust Self-Supervised Speaker Verification](https://mlsanity.com/paper/2505.14561/) (2025)
- [Towards noise robust trigger-word detection with contrastive learning pre-task for fast on-boarding of new trigger-words](https://mlsanity.com/paper/2111.03971/) (2021)
- [Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification](https://mlsanity.com/paper/2112.08929/) (2021)
- [Dynamic Recognition of Speakers for Consent Management by Contrastive Embedding Replay](https://mlsanity.com/paper/2205.08459/) (2022)
- [Cluster-Guided Unsupervised Domain Adaptation for Deep Speaker Embedding](https://mlsanity.com/paper/2303.15944/) (2023)
- [Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement](https://mlsanity.com/paper/2306.08406/) (2023)
- [Self-supervised learning with diffusion-based multichannel speech enhancement for speaker verification under noisy conditions](https://mlsanity.com/paper/2307.02244/) (2023)
- [Speech Representation Learning Revisited: The Necessity of Separate Learnable Parameters and Robust Data Augmentation](https://mlsanity.com/paper/2408.10557/) (2024)
- [Self-Supervised Learning for Speaker Recognition: A study and review](https://mlsanity.com/paper/2602.10829/) (2026)
- [Learning task-specific subspaces via interventional post-training of speech foundation models](https://mlsanity.com/paper/2606.17967/) (2026)

## Abstract (arXiv)

In this study, we investigate self-supervised representation learning for speaker verification (SV). First, we examine a simple contrastive learning approach (SimCLR) with a momentum contrastive (MoCo) learning framework, where the MoCo speaker embedding system utilizes a queue to maintain a large set of negative examples. We show that better speaker embeddings can be learned by momentum contrastive learning. Next, alternative augmentation strategies are explored to normalize extrinsic speaker variabilities of two random segments from the same speech utterance. Specifically, augmentation in the waveform largely improves the speaker representations for SV tasks. The proposed MoCo speaker embedding is further improved when a prototypical memory bank is introduced, which encourages the speaker embeddings to be closer to their assigned prototypes with an intermediate clustering step. In addition, we generalize the self-supervised framework to a semi-supervised scenario where only a small portion of the data is labeled. Comprehensive experiments on the Voxceleb dataset demonstrate that our proposed self-supervised approach achieves competitive performance compared with existing techniques, and can approach fully supervised results with partially labeled data.