# Multi-View Self-Attention Based Transformer for Speaker Recognition

arXiv: https://arxiv.org/abs/2110.05036 · HTML: https://mlsanity.com/paper/2110.05036/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System](https://mlsanity.com/paper/1804.05160/) (2018)
- [Self-attention encoding and pooling for speaker recognition](https://mlsanity.com/paper/2008.01077/) (2020)
- [Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretraining](https://mlsanity.com/paper/1912.06813/) (2019)
- [A Configurable Multilingual Model is All You Need to Recognize All Languages](https://mlsanity.com/paper/2107.05876/) (2021)
- [Delving into VoxCeleb: environment invariant speaker recognition](https://mlsanity.com/paper/1910.11238/) (2019)
- [Serialized Multi-Layer Multi-Head Attention for Neural Speaker Embedding](https://mlsanity.com/paper/2107.06493/) (2021)
- [Double Multi-Head Attention for Speaker Verification](https://mlsanity.com/paper/2007.13199/) (2020)
- [T-vectors: Weakly Supervised Speaker Identification Using Hierarchical Transformer Model](https://mlsanity.com/paper/2010.16071/) (2020)
- [MACCIF-TDNN: Multi aspect aggregation of channel and context interdependence features in TDNN-based speaker verification](https://mlsanity.com/paper/2107.03104/) (2021)

## Built on by (most cited first, 8 of 8)

- [MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning](https://mlsanity.com/paper/2307.07951/) (2023)
- [Spatio-Temporal Representation Learning Enhanced Source Cell-phone Recognition from Speech Recordings](https://mlsanity.com/paper/2208.12753/) (2022)
- [Large-scale learning of generalised representations for speaker recognition](https://mlsanity.com/paper/2210.10985/) (2022)
- [Improving Transformer-based Networks With Locality For Automatic Speaker Verification](https://mlsanity.com/paper/2302.08639/) (2023)
- [Parameter-Free Attentive Scoring for Speaker Verification](https://mlsanity.com/paper/2203.05642/) (2022)
- [Comparative Analysis of Modality Fusion Approaches for Audio-Visual Person Identification and Verification](https://mlsanity.com/paper/2409.00562/) (2024)
- [Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems](https://mlsanity.com/paper/2111.03842/) (2021)
- [MECT: Mixture of Experts with CNN-Transformer Network for Speaker verification](https://mlsanity.com/paper/2609.24061/) (2026)

## Abstract (arXiv)

Initially developed for natural language processing (NLP), Transformer model is now widely used for speech processing tasks such as speaker recognition, due to its powerful sequence modeling capabilities. However, conventional self-attention mechanisms are originally designed for modeling textual sequence without considering the characteristics of speech and speaker modeling. Besides, different Transformer variants for speaker recognition have not been well studied. In this work, we propose a novel multi-view self-attention mechanism and present an empirical study of different Transformer variants with or without the proposed attention mechanism for speaker recognition. Specifically, to balance the capabilities of capturing global dependencies and modeling the locality, we propose a multi-view self-attention mechanism for speaker Transformer, in which different attention heads can attend to different ranges of the receptive field. Furthermore, we introduce and compare five Transformer variants with different network architectures, embedding locations, and pooling methods to learn speaker embeddings. Experimental results on the VoxCeleb1 and VoxCeleb2 datasets show that the proposed multi-view self-attention mechanism achieves improvement in the performance of speaker recognition, and the proposed speaker Transformer network attains excellent results compared with state-of-the-art models.