# Improving Transformer-based Networks With Locality For Automatic Speaker Verification

arXiv: https://arxiv.org/abs/2302.08639 · HTML: https://mlsanity.com/paper/2302.08639/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 28 of 28)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System](https://mlsanity.com/paper/1804.05160/) (2018)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [Self-attention encoding and pooling for speaker recognition](https://mlsanity.com/paper/2008.01077/) (2020)
- [Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?](https://mlsanity.com/paper/2204.12765/) (2022)
- [Semi-Supervised Contrastive Learning with Generalized Contrastive Loss and Its Application to Speaker Recognition](https://mlsanity.com/paper/2006.04326/) (2020)
- [Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularization](https://mlsanity.com/paper/2112.04459/) (2021)
- [ResNeXt and Res2Net Structures for Speaker Verification](https://mlsanity.com/paper/2007.02480/) (2020)
- [DEAAN: Disentangled Embedding and Adversarial Adaptation Network for Robust Speaker Representation Learning](https://mlsanity.com/paper/2012.06896/) (2020)
- [S-vectors and TESA: Speaker Embeddings and a Speaker Authenticator Based on Transformer Encoder](https://mlsanity.com/paper/2008.04659/) (2020)
- [Multi-View Self-Attention Based Transformer for Speaker Recognition](https://mlsanity.com/paper/2110.05036/) (2021)
- [Generative Adversarial Speaker Embedding Networks for Domain Robust End-to-End Speaker Verification](https://mlsanity.com/paper/1811.03063/) (2018)
- [Deep Speaker Embedding Learning with Multi-Level Pooling for Text-Independent Speaker Verification](https://mlsanity.com/paper/1902.07821/) (2019)
- [Open-set Short Utterance Forensic Speaker Verification using Teacher-Student Network with Explicit Inductive Bias](https://mlsanity.com/paper/2009.09556/) (2020)
- [Multi-Frequency Information Enhanced Channel Attention Module for Speaker Representation Learning](https://mlsanity.com/paper/2207.04540/) (2022)
- [DT-SV: A Transformer-based Time-domain Approach for Speaker Verification](https://mlsanity.com/paper/2205.13249/) (2022)

## Built on by (most cited first, 3 of 3)

- [Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verification](https://mlsanity.com/paper/2403.00293/) (2024)
- [Toward Improving Synthetic Audio Spoofing Detection Robustness via Meta-Learning and Disentangled Training With Adversarial Examples](https://mlsanity.com/paper/2408.13341/) (2024)
- [UniPET-SPK: A Unified Framework for Parameter-Efficient Tuning of Pre-trained Speech Models for Robust Speaker Verification](https://mlsanity.com/paper/2501.16542/) (2025)

## Abstract (arXiv)

Recently, Transformer-based architectures have been explored for speaker embedding extraction. Although the Transformer employs the self-attention mechanism to efficiently model the global interaction between token embeddings, it is inadequate for capturing short-range local context, which is essential for the accurate extraction of speaker information. In this study, we enhance the Transformer with the enhanced locality modeling in two directions. First, we propose the Locality-Enhanced Conformer (LE-Confomer) by introducing depth-wise convolution and channel-wise attention into the Conformer blocks. Second, we present the Speaker Swin Transformer (SST) by adapting the Swin Transformer, originally proposed for vision tasks, into speaker embedding network. We evaluate the proposed approaches on the VoxCeleb datasets and a large-scale Microsoft internal multilingual (MS-internal) dataset. The proposed models achieve 0.75% EER on VoxCeleb 1 test set, outperforming the previously proposed Transformer-based models and CNN-based models, such as ResNet34 and ECAPA-TDNN. When trained on the MS-internal dataset, the proposed models achieve promising results with 14.6% relative reduction in EER over the Res2Net50 model.