# On Training Targets and Activation Functions for Deep Representation Learning in Text-Dependent Speaker Verification

arXiv: https://arxiv.org/abs/2201.06426 · HTML: https://mlsanity.com/paper/2201.06426/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 68 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 19 of 19)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder](https://mlsanity.com/paper/1603.00982/) (2016)
- [rVAD: An Unsupervised Segment-Based Robust Voice Activity Detection Method](https://mlsanity.com/paper/1906.03588/) (2019)
- [Short-duration Speaker Verification (SdSV) Challenge 2021: the Challenge Evaluation Plan](https://mlsanity.com/paper/1912.06311/) (2019)
- [OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer](https://mlsanity.com/paper/2004.09033/) (2020)
- [Time-Contrastive Learning Based Deep Bottleneck Features for Text-Dependent Speaker Verification](https://mlsanity.com/paper/1905.04554/) (2019)
- [Low-Dimensional Bottleneck Features for On-Device Continuous Speech Recognition](https://mlsanity.com/paper/1811.00006/) (2018)
- [Vocal Tract Length Perturbation for Text-Dependent Speaker Verification with Autoregressive Prediction Coding](https://mlsanity.com/paper/2011.12536/) (2020)
- [Fusion of Embeddings Networks for Robust Combination of Text Dependent and Independent Speaker Recognition](https://mlsanity.com/paper/2106.10169/) (2021)
- [Phoneme-aware and Channel-wise Attentive Learning for Text DependentSpeaker Verification](https://mlsanity.com/paper/2106.13514/) (2021)
- [Incorporating Pass-Phrase Dependent Background Models for Text-Dependent Speaker Verification](https://mlsanity.com/paper/1611.06423/) (2016)
- [UIAI System for Short-Duration Speaker Verification Challenge 2020](https://mlsanity.com/paper/2007.13118/) (2020)
- [Poformer: A simple pooling transformer for speaker verification](https://mlsanity.com/paper/2110.04692/) (2021)

## Abstract (arXiv)

Deep representation learning has gained significant momentum in advancing text-dependent speaker verification (TD-SV) systems. When designing deep neural networks (DNN) for extracting bottleneck features, key considerations include training targets, activation functions, and loss functions. In this paper, we systematically study the impact of these choices on the performance of TD-SV. For training targets, we consider speaker identity, time-contrastive learning (TCL) and auto-regressive prediction coding with the first being supervised and the last two being self-supervised. Furthermore, we study a range of loss functions when speaker identity is used as the training target. With regard to activation functions, we study the widely used sigmoid function, rectified linear unit (ReLU), and Gaussian error linear unit (GELU). We experimentally show that GELU is able to reduce the error rates of TD-SV significantly compared to sigmoid, irrespective of training target. Among the three training targets, TCL performs the best. Among the various loss functions, cross entropy, joint-softmax and focal loss functions outperform the others. Finally, score-level fusion of different systems is also able to reduce the error rates. Experiments are conducted on the RedDots 2016 challenge database for TD-SV using short utterances. For the speaker classifications, the well-known Gaussian mixture model-universal background model (GMM-UBM) and i-vector techniques are used.