# Serialized Multi-Layer Multi-Head Attention for Neural Speaker Embedding

arXiv: https://arxiv.org/abs/2107.06493 · HTML: https://mlsanity.com/paper/2107.06493/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)
- [Self-attention encoding and pooling for speaker recognition](https://mlsanity.com/paper/2008.01077/) (2020)
- [Self Multi-Head Attention for Speaker Recognition](https://mlsanity.com/paper/1906.09890/) (2019)
- [On deep speaker embeddings for text-independent speaker recognition](https://mlsanity.com/paper/1804.10080/) (2018)
- [Exploring a Unified Attention-Based Pooling Framework for Speaker Verification](https://mlsanity.com/paper/1808.07120/) (2018)

## Built on by (most cited first, 4 of 4)

- [Multi-View Self-Attention Based Transformer for Speaker Recognition](https://mlsanity.com/paper/2110.05036/) (2021)
- [Large-scale learning of generalised representations for speaker recognition](https://mlsanity.com/paper/2210.10985/) (2022)
- [DT-SV: A Transformer-based Time-domain Approach for Speaker Verification](https://mlsanity.com/paper/2205.13249/) (2022)
- [Attention-based conditioning methods using variable frame rate for style-robust speaker verification](https://mlsanity.com/paper/2206.13680/) (2022)

## Abstract (arXiv)

This paper proposes a serialized multi-layer multi-head attention for neural speaker embedding in text-independent speaker verification. In prior works, frame-level features from one layer are aggregated to form an utterance-level representation. Inspired by the Transformer network, our proposed method utilizes the hierarchical architecture of stacked self-attention mechanisms to derive refined features that are more correlated with speakers. Serialized attention mechanism contains a stack of self-attention modules to create fixed-dimensional representations of speakers. Instead of utilizing multi-head attention in parallel, the proposed serialized multi-layer multi-head attention is designed to aggregate and propagate attentive statistics from one layer to the next in a serialized manner. In addition, we employ an input-aware query for each utterance with the statistics pooling. With more layers stacked, the neural network can learn more discriminative speaker embeddings. Experiment results on VoxCeleb1 dataset and SITW dataset show that our proposed method outperforms other baseline methods, including x-vectors and other x-vectors + conventional attentive pooling approaches by 9.7% in EER and 8.1% in DCF0.01.