# Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification

arXiv: https://arxiv.org/abs/2309.14838 · HTML: https://mlsanity.com/paper/2309.14838/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 16 of 16)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Multi-task Self-Supervised Learning for Human Activity Detection](https://mlsanity.com/paper/1907.11879/) (2019)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [BUT System Description to VoxCeleb Speaker Recognition Challenge 2019](https://mlsanity.com/paper/1910.12592/) (2019)
- [Self-supervised Speaker Recognition with Loss-gated Learning](https://mlsanity.com/paper/2110.03869/) (2021)
- [Estimation of speaker age and height from speech signal using bi-encoder transformer mixture model](https://mlsanity.com/paper/2203.11774/) (2022)
- [On the Efficiency of Integrating Self-supervised Learning and Meta-learning for User-defined Few-shot Keyword Spotting](https://mlsanity.com/paper/2204.00352/) (2022)
- [One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification](https://mlsanity.com/paper/2305.17394/) (2023)
- [Training speaker recognition systems with limited data](https://mlsanity.com/paper/2203.14688/) (2022)
- [Label-free Knowledge Distillation with Contrastive Loss for Light-weight Speaker Recognition](https://mlsanity.com/paper/2212.03090/) (2022)
- [Distilling Multi-Level X-vector Knowledge for Small-footprint Speaker Verification](https://mlsanity.com/paper/2303.01125/) (2023)

## Built on by (most cited first, 4 of 4)

- [Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detection](https://mlsanity.com/paper/2406.17376/) (2024)
- [Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems](https://mlsanity.com/paper/2407.03645/) (2024)
- [DISPATCH: Distilling Selective Patches for Speech Enhancement](https://mlsanity.com/paper/2509.15922/) (2025)
- [Triage knowledge distillation for speaker verification](https://mlsanity.com/paper/2601.14699/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) is used to enhance automatic speaker verification performance by ensuring consistency between large teacher networks and lightweight student networks at the embedding level or label level. However, the conventional label-level KD overlooks the significant knowledge from non-target speakers, particularly their classification probabilities, which can be crucial for automatic speaker verification. In this paper, we first demonstrate that leveraging a larger number of training non-target speakers improves the performance of automatic speaker verification models. Inspired by this finding about the importance of non-target speakers' knowledge, we modified the conventional label-level KD by disentangling and emphasizing the classification probabilities of non-target speakers during knowledge distillation. The proposed method is applied to three different student model architectures and achieves an average of 13.67% improvement in EER on the VoxCeleb dataset compared to embedding-level and conventional label-level KD methods.