# Contrastive Knowledge Distillation for Embedding Refinement in Personalized Speech Enhancement

arXiv: https://arxiv.org/abs/2601.16235 · HTML: https://mlsanity.com/paper/2601.16235/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 20 of 20)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [SpEx+: A Complete Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2005.04686/) (2020)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio](https://mlsanity.com/paper/2205.05474/) (2022)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/1912.02522/) (2019)
- [Personalized PercepNet: Real-time, Low-complexity Target Voice Separation and Enhancement](https://mlsanity.com/paper/2106.04129/) (2021)
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022)
- [ICASSP 2021 Deep Noise Suppression Challenge: Decoupling Magnitude and Phase Optimization with a Two-Stage Deep Network](https://mlsanity.com/paper/2102.04198/) (2021)
- [Towards Lightweight Applications: Asymmetric Enroll-Verify Structure for Speaker Verification](https://mlsanity.com/paper/2110.04438/) (2021)
- [Robust Speaker Extraction Network Based on Iterative Refined Adaptation](https://mlsanity.com/paper/2011.02102/) (2020)
- [Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation](https://mlsanity.com/paper/2210.12635/) (2022)
- [Multi-stage Speaker Extraction with Utterance and Frame-Level Reference Signals](https://mlsanity.com/paper/2011.09624/) (2020)
- [A lightweight dual-stage framework for personalized speech enhancement based on DeepFilterNet2](https://mlsanity.com/paper/2404.08022/) (2024)

## Abstract (arXiv)

Personalized speech enhancement (PSE) has shown convincing results when it comes to extracting a known target voice among interfering ones. The corresponding systems usually incorporate a representation of the target voice within the enhancement system, which is extracted from an enrollment clip of the target voice with upstream models. Those models are generally heavy as the speaker embedding's quality directly affects PSE performances. Yet, embeddings generated beforehand cannot account for the variations of the target voice during inference time. In this paper, we propose to perform on-thefly refinement of the speaker embedding using a tiny speaker encoder. We first introduce a novel contrastive knowledge distillation methodology in order to train a 150k-parameter encoder from complex embeddings. We then use this encoder within the enhancement system during inference and show that the proposed method greatly improves PSE performances while maintaining a low computational load.