# Multi-query multi-head attention pooling and Inter-topK penalty for speaker verification

arXiv: https://arxiv.org/abs/2110.05042 · HTML: https://mlsanity.com/paper/2110.05042/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2021.

## Built on by (most cited first, 9 of 9)

- [USTC-KXDIGIT System Description for ASVspoof5 Challenge](https://mlsanity.com/paper/2409.01695/) (2024)
- [The SpeakIn Speaker Verification System for Far-Field Speaker Verification Challenge 2022](https://mlsanity.com/paper/2209.11625/) (2022)
- [UNISOUND System for VoxCeleb Speaker Recognition Challenge 2023](https://mlsanity.com/paper/2308.12526/) (2023)
- [STAMP: Spatial-Temporal Adapter with Multi-Head Pooling](https://mlsanity.com/paper/2511.10848/) (2025)
- [Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification](https://mlsanity.com/paper/2512.22148/) (2025)
- [Progressive Sub-Graph Clustering Algorithm for Semi-Supervised Domain Adaptation Speaker Verification](https://mlsanity.com/paper/2305.12703/) (2023)
- [ChinaTelecom System Description to VoxCeleb Speaker Recognition Challenge 2023](https://mlsanity.com/paper/2308.08181/) (2023)
- [Disentangling Age and Identity with a Mutual Information Minimization Approach for Cross-Age Speaker Verification](https://mlsanity.com/paper/2409.15974/) (2024)
- [Hear: Hierarchically Enhanced Aesthetic Representations For Multidimensional Music Evaluation](https://mlsanity.com/paper/2511.18869/) (2025)

## Abstract (arXiv)

This paper describes the multi-query multi-head attention (MQMHA) pooling and inter-topK penalty methods which were first proposed in our submitted system description for VoxCeleb speaker recognition challenge (VoxSRC) 2021. Most multi-head attention pooling mechanisms either attend to the whole feature through multiple heads or attend to several split parts of the whole feature. Our proposed MQMHA combines both these two mechanisms and gain more diversified information. The margin-based softmax loss functions are commonly adopted to obtain discriminative speaker representations. To further enhance the inter-class discriminability, we propose a method that adds an extra inter-topK penalty on some confused speakers. By adopting both the MQMHA and inter-topK penalty, we achieved state-of-the-art performance in all of the public VoxCeleb test sets.