# Self-attention encoding and pooling for speaker recognition

arXiv: https://arxiv.org/abs/2008.01077 · HTML: https://mlsanity.com/paper/2008.01077/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Aug 2020.

## Built on by (most cited first, 30 of 33)

- [StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles](https://mlsanity.com/paper/2301.01081/) (2023)
- [SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation](https://mlsanity.com/paper/2205.08180/) (2022)
- [Say Anything with Any Style](https://mlsanity.com/paper/2403.06363/) (2024)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [SpeechNet: A Universal Modularized Model for Speech Processing Tasks](https://mlsanity.com/paper/2105.03070/) (2021)
- [Membership Inference Attacks Against Self-supervised Speech Models](https://mlsanity.com/paper/2111.05113/) (2021)
- [DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models](https://mlsanity.com/paper/2312.09767/) (2023)
- [StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads](https://mlsanity.com/paper/2409.09292/) (2024)
- [Multi-View Self-Attention Based Transformer for Speaker Recognition](https://mlsanity.com/paper/2110.05036/) (2021)
- [Improving Speech Enhancement through Fine-Grained Speech Characteristics](https://mlsanity.com/paper/2207.00237/) (2022)
- [Improving End-to-End SLU performance with Prosodic Attention and Distillation](https://mlsanity.com/paper/2305.08067/) (2023)
- [Serialized Multi-Layer Multi-Head Attention for Neural Speaker Embedding](https://mlsanity.com/paper/2107.06493/) (2021)
- [Large-scale learning of generalised representations for speaker recognition](https://mlsanity.com/paper/2210.10985/) (2022)
- [L-TUNING: Synchronized Label Tuning for Prompt and Prefix in LLMs](https://mlsanity.com/paper/2402.01643/) (2023)
- [A Concept-based Interpretable Model for the Diagnosis of Choroid Neoplasias using Multimodal Data](https://mlsanity.com/paper/2403.05606/) (2024)
- [EfficientTDNN: Efficient Architecture Search for Speaker Recognition](https://mlsanity.com/paper/2103.13581/) (2021)
- [Improving Transformer-based Networks With Locality For Automatic Speaker Verification](https://mlsanity.com/paper/2302.08639/) (2023)
- [Keyword-driven Retrieval-Augmented Large Language Models for Cold-start User Recommendations](https://mlsanity.com/paper/2405.19612/) (2024)
- [MSP-Podcast SER Challenge 2024: L'antenne du Ventoux Multimodal Self-Supervised Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2407.05746/) (2024)
- [PDAF: A Phonetic Debiasing Attention Framework For Speaker Verification](https://mlsanity.com/paper/2409.05799/) (2024)
- [Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning](https://mlsanity.com/paper/2501.14680/) (2025)
- [The NTNU System at the S&I Challenge 2025 SLA Open Track](https://mlsanity.com/paper/2506.05121/) (2025)
- [Parameter-Free Attentive Scoring for Speaker Verification](https://mlsanity.com/paper/2203.05642/) (2022)
- [Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings](https://mlsanity.com/paper/2210.12857/) (2022)
- [Automatic Modulation Classification with Deep Neural Networks](https://mlsanity.com/paper/2301.11773/) (2023)
- [Improved Cross-Lingual Transfer Learning For Automatic Speech Translation](https://mlsanity.com/paper/2306.00789/) (2023)
- [Speaker embeddings by modeling channel-wise correlations](https://mlsanity.com/paper/2104.02571/) (2021)
- [Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition](https://mlsanity.com/paper/2210.16642/) (2022)
- [An Effective Transformer-based Contextual Model and Temporal Gate Pooling for Speaker Identification](https://mlsanity.com/paper/2308.11241/) (2023)
- [From Black Box to Biomarker: Sparse Autoencoders for Interpreting Speech Models of Parkinson's Disease](https://mlsanity.com/paper/2507.16836/) (2025)

## Abstract (arXiv)

The computing power of mobile devices limits the end-user applications in terms of storage size, processing, memory and energy consumption. These limitations motivate researchers for the design of more efficient deep models. On the other hand, self-attention networks based on Transformer architecture have attracted remarkable interests due to their high parallelization capabilities and strong performance on a variety of Natural Language Processing (NLP) applications. Inspired by the Transformer, we propose a tandem Self-Attention Encoding and Pooling (SAEP) mechanism to obtain a discriminative speaker embedding given non-fixed length speech utterances. SAEP is a stack of identical blocks solely relied on self-attention and position-wise feed-forward networks to create vector representation of speakers. This approach encodes short-term speaker spectral features into speaker embeddings to be used in text-independent speaker verification. We have evaluated this approach on both VoxCeleb1 & 2 datasets. The proposed architecture is able to outperform the baseline x-vector, and shows competitive performance to some other benchmarks based on convolutions, with a significant reduction in model size. It employs 94%, 95%, and 73% less parameters compared to ResNet-34, ResNet-50, and x-vector, respectively. This indicates that the proposed fully attention based architecture is more efficient in extracting time-invariant features from speaker utterances.