# Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification

arXiv: https://arxiv.org/abs/2303.01125 · HTML: https://mlsanity.com/paper/2303.01125/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 71 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [Attentive Statistics Pooling for Deep Speaker Embedding](https://mlsanity.com/paper/1803.10963/) (2018)
- [Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation](https://mlsanity.com/paper/2005.08575/) (2020)
- [Speaker Recognition Based on Deep Learning: An Overview](https://mlsanity.com/paper/2012.00931/) (2020)
- [A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification](https://mlsanity.com/paper/1804.00385/) (2018)
- [Building Proactive Voice Assistants: When and How (not) to Interact](https://mlsanity.com/paper/2005.01322/) (2020)
- [Extremely Low Footprint End-to-End ASR System for Smart Device](https://mlsanity.com/paper/2104.05784/) (2021)
- [Towards Lightweight Applications: Asymmetric Enroll-Verify Structure for Speaker Verification](https://mlsanity.com/paper/2110.04438/) (2021)
- [Baselines and Protocols for Household Speaker Recognition](https://mlsanity.com/paper/2205.00288/) (2022)
- [AM-MobileNet1D: A Portable Model for Speaker Recognition](https://mlsanity.com/paper/2004.00132/) (2020)
- [Binary Neural Network for Speaker Verification](https://mlsanity.com/paper/2104.02306/) (2021)

## Built on by (most cited first, 1 of 1)

- [Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification](https://mlsanity.com/paper/2309.14838/) (2023)

## Abstract (arXiv)

Even though deep speaker models have demonstrated impressive accuracy in speaker verification tasks, this often comes at the expense of increased model size and computation time, presenting challenges for deployment in resource-constrained environments. Our research focuses on addressing this limitation through the development of small footprint deep speaker embedding extraction using knowledge distillation. While previous work in this domain has concentrated on speaker embedding extraction at the utterance level, our approach involves amalgamating embeddings from different levels of the x-vector model (teacher network) to train a compact student network. The results highlight the significance of frame-level information, with the student models exhibiting a remarkable size reduction of 85%-91% compared to their teacher counterparts, depending on the size of the teacher embeddings. Notably, by concatenating teacher embeddings, we achieve student networks that maintain comparable performance to the teacher while enjoying a substantial 75% reduction in model size. These findings and insights extend to other x-vector variants, underscoring the broad applicability of our approach.