# Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?

arXiv: https://arxiv.org/abs/2204.12765 · HTML: https://mlsanity.com/paper/2204.12765/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training](https://mlsanity.com/paper/2110.05752/) (2021)
- [The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification](https://mlsanity.com/paper/2010.11255/) (2020)
- [The SpeakIn System for VoxCeleb Speaker Recognition Challange 2021](https://mlsanity.com/paper/2109.01989/) (2021)
- [Self-Supervised Learning for speech recognition with Intermediate layer supervision](https://mlsanity.com/paper/2112.08778/) (2021)
- [ResNeXt and Res2Net Structures for Speaker Verification](https://mlsanity.com/paper/2007.02480/) (2020)

## Built on by (most cited first, 27 of 27)

- [Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora](https://mlsanity.com/paper/2210.02595/) (2022)
- [Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech](https://mlsanity.com/paper/2210.11723/) (2022)
- [Coding Speech through Vocal Tract Kinematics](https://mlsanity.com/paper/2406.12998/) (2024)
- [Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model](https://mlsanity.com/paper/2304.11976/) (2023)
- [On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding](https://mlsanity.com/paper/2210.05291/) (2022)
- [Speech separation with large-scale self-supervised learning](https://mlsanity.com/paper/2211.05172/) (2022)
- [Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition](https://mlsanity.com/paper/2308.08713/) (2023)
- [Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters](https://mlsanity.com/paper/2401.05111/) (2024)
- [Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models](https://mlsanity.com/paper/2406.02285/) (2024)
- [Layer-wise Investigation of Large-Scale Self-Supervised Music Representation Models](https://mlsanity.com/paper/2505.16306/) (2025)
- [Improving Transformer-based Networks With Locality For Automatic Speaker Verification](https://mlsanity.com/paper/2302.08639/) (2023)
- [Voice Impression Control in Zero-Shot TTS](https://mlsanity.com/paper/2506.05688/) (2025)
- [Semantic enrichment towards efficient speech representations](https://mlsanity.com/paper/2307.01323/) (2023)
- [Lightweight Zero-shot Text-to-Speech with Mixture of Adapters](https://mlsanity.com/paper/2407.01291/) (2024)
- [Layer-aware TDNN: Speaker Recognition Using Multi-Layer Features from Pre-Trained Models](https://mlsanity.com/paper/2409.07770/) (2024)
- [CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing](https://mlsanity.com/paper/2412.04425/) (2024)
- [Multi-interaction TTS toward professional recording reproduction](https://mlsanity.com/paper/2507.00808/) (2025)
- [DAFMSVC: One-Shot Singing Voice Conversion with Dual Attention Mechanism and Flow Matching](https://mlsanity.com/paper/2508.05978/) (2025)
- [Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models](https://mlsanity.com/paper/2510.02569/) (2025)
- [A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding](https://mlsanity.com/paper/2406.12141/) (2024)
- [Singing Voice Conversion with Accompaniment Using Self-Supervised Representation-Based Melody Features](https://mlsanity.com/paper/2502.04722/) (2025)
- [Training Articulatory Inversion Models for Interspeaker Consistency](https://mlsanity.com/paper/2505.20529/) (2025)
- [Investigation for Relative Voice Impression Estimation](https://mlsanity.com/paper/2602.14172/) (2026)
- [Beyond Short Segments : Expanding Speaker Embeddings with Vector Archives](https://mlsanity.com/paper/2609.25007/) (2026)
- [An Empirical Analysis of Speech Self-Supervised Learning at Multiple Resolutions](https://mlsanity.com/paper/2410.23955/) (2024)
- [Self-Supervised Learning for Speaker Recognition: A study and review](https://mlsanity.com/paper/2602.10829/) (2026)
- [LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation](https://mlsanity.com/paper/2605.00777/) (2026)

## Abstract (arXiv)

Recently, self-supervised learning (SSL) has demonstrated strong performance in speaker recognition, even if the pre-training objective is designed for speech recognition. In this paper, we study which factor leads to the success of self-supervised learning on speaker-related tasks, e.g. speaker verification (SV), through a series of carefully designed experiments. Our empirical results on the Voxceleb-1 dataset suggest that the benefit of SSL to SV task is from a combination of mask speech prediction loss, data scale, and model size, while the SSL quantizer has a minor impact. We further employ the integrated gradients attribution method and loss landscape visualization to understand the effectiveness of self-supervised learning for speaker recognition performance.