# TMS: A Temporal Multi-scale Backbone Design for Speaker Embedding

arXiv: https://arxiv.org/abs/2203.09098 · HTML: https://mlsanity.com/paper/2203.09098/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors](https://mlsanity.com/paper/2005.09921/) (2020)
- [CN-CELEB: a challenging Chinese speaker recognition dataset](https://mlsanity.com/paper/1911.01799/) (2019)
- [CN-Celeb: multi-genre speaker recognition](https://mlsanity.com/paper/2012.12468/) (2020)
- [The IDLAB VoxCeleb Speaker Recognition Challenge 2020 System Description](https://mlsanity.com/paper/2010.12468/) (2020)
- [SpeakerNet: 1D Depth-wise Separable Convolutional Network for Text-Independent Speaker Recognition and Verification](https://mlsanity.com/paper/2010.12653/) (2020)
- [The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.12731/) (2020)
- [HS-ResNet: Hierarchical-Split Block on Convolutional Neural Network](https://mlsanity.com/paper/2010.07621/) (2020)
- [Deep Normalization for Speaker Vectors](https://mlsanity.com/paper/2004.04095/) (2020)
- [Rep Works in Speaker Verification](https://mlsanity.com/paper/2110.09720/) (2021)
- [Deep multi-metric learning for text-independent speaker verification](https://mlsanity.com/paper/2007.10479/) (2020)
- [CS-Rep: Making Speaker Verification Networks Embracing Re-parameterization](https://mlsanity.com/paper/2110.13465/) (2021)

## Built on by (most cited first, 2 of 2)

- [Frequency and Multi-Scale Selective Kernel Attention for Speaker Verification](https://mlsanity.com/paper/2204.01005/) (2022)
- [Cluster-Guided Unsupervised Domain Adaptation for Deep Speaker Embedding](https://mlsanity.com/paper/2303.15944/) (2023)

## Abstract (arXiv)

Speaker embedding is an important front-end module to explore discriminative speaker features for many speech applications where speaker information is needed. Current SOTA backbone networks for speaker embedding are designed to aggregate multi-scale features from an utterance with multi-branch network architectures for speaker representation. However, naively adding many branches of multi-scale features with the simple fully convolutional operation could not efficiently improve the performance due to the rapid increase of model parameters and computational complexity. Therefore, in the most current state-of-the-art network architectures, only a few branches corresponding to a limited number of temporal scales could be designed for speaker embeddings. To address this problem, in this paper, we propose an effective temporal multi-scale (TMS) model where multi-scale branches could be efficiently designed in a speaker embedding network almost without increasing computational costs. The new model is based on the conventional TDNN, where the network architecture is smartly separated into two modeling operators: a channel-modeling operator and a temporal multi-branch modeling operator. Adding temporal multi-scale in the temporal multi-branch operator needs only a little bit increase of the number of parameters, and thus save more computational budget for adding more branches with large temporal scales. Moreover, in the inference stage, we further developed a systemic re-parameterization method to convert the TMS-based model into a single-path-based topology in order to increase inference speed. We investigated the performance of the new TMS method for automatic speaker verification (ASV) on in-domain and out-of-domain conditions. Results show that the TMS-based model obtained a significant increase in the performance over the SOTA ASV models, meanwhile, had a faster inference speed.