# Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems

arXiv: https://arxiv.org/abs/2111.03842 · HTML: https://mlsanity.com/paper/2111.03842/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 6 Nov 2021.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Large Memory Layers with Product Keys](https://mlsanity.com/paper/1907.05242/) (2019)
- [Bayesian Dark Knowledge](https://mlsanity.com/paper/1506.04416/) (2015)
- [Self Multi-Head Attention for Speaker Recognition](https://mlsanity.com/paper/1906.09890/) (2019)
- [A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: the DeepMine Database](https://mlsanity.com/paper/1912.03627/) (2019)
- [Short-duration Speaker Verification (SdSV) Challenge 2021: the Challenge Evaluation Plan](https://mlsanity.com/paper/1912.06311/) (2019)
- [Multi-View Self-Attention Based Transformer for Speaker Recognition](https://mlsanity.com/paper/2110.05036/) (2021)
- [A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification](https://mlsanity.com/paper/1804.00385/) (2018)
- [Speaker-Utterance Dual Attention for Speaker and Utterance Verification](https://mlsanity.com/paper/2008.08901/) (2020)
- [Optimization of the Area Under the ROC Curve using Neural Network Supervectors for Text-Dependent Speaker Verification](https://mlsanity.com/paper/1901.11332/) (2019)
- [Differentiable Supervector Extraction for Encoding Speaker and Phrase Information in Text Dependent Speaker Verification](https://mlsanity.com/paper/1812.09484/) (2018)

## Built on by (most cited first, 1 of 1)

- [DISPATCH: Distilling Selective Patches for Speech Enhancement](https://mlsanity.com/paper/2509.15922/) (2025)

## Abstract (arXiv)

This paper explores three novel approaches to improve the performance of speaker verification (SV) systems based on deep neural networks (DNN) using Multi-head Self-Attention (MSA) mechanisms and memory layers. Firstly, we propose the use of a learnable vector called Class token to replace the average global pooling mechanism to extract the embeddings. Unlike global average pooling, our proposal takes into account the temporal structure of the input what is relevant for the text-dependent SV task. The class token is concatenated to the input before the first MSA layer, and its state at the output is used to predict the classes. To gain additional robustness, we introduce two approaches. First, we have developed a Bayesian estimation of the class token. Second, we have added a distilled representation token for training a teacher-student pair of networks using the Knowledge Distillation (KD) philosophy, which is combined with the class token. This distillation token is trained to mimic the predictions from the teacher network, while the class token replicates the true label. All the strategies have been tested on the RSR2015-Part II and DeepMine-Part 1 databases for text-dependent SV, providing competitive results compared to the same architecture using the average pooling mechanism to extract average embeddings.