# Breaking the trade-off in personalized speech enhancement with cross-task knowledge distillation

arXiv: https://arxiv.org/abs/2211.02944 · HTML: https://mlsanity.com/paper/2211.02944/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 5 Nov 2022.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 11 of 11)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam](https://mlsanity.com/paper/2001.08378/) (2020)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [Personalized PercepNet: Real-time, Low-complexity Target Voice Separation and Enhancement](https://mlsanity.com/paper/2106.04129/) (2021)
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022)
- [X-TaSNet: Robust and Accurate Time-Domain Speaker Extraction Network](https://mlsanity.com/paper/2010.12766/) (2020)
- [On training targets for noise-robust voice activity detection](https://mlsanity.com/paper/2102.07445/) (2021)
- [One model to enhance them all: array geometry agnostic multi-channel personalized speech enhancement](https://mlsanity.com/paper/2110.10330/) (2021)
- [Listen only to me! How well can target speech extraction handle false alarms?](https://mlsanity.com/paper/2204.04811/) (2022)

## Abstract (arXiv)

Personalized speech enhancement (PSE) models achieve promising results compared with unconditional speech enhancement models due to their ability to remove interfering speech in addition to background noise. Unlike unconditional speech enhancement, causal PSE models may occasionally remove the target speech by mistake. The PSE models also tend to leak interfering speech when the target speaker is silent for an extended period. We show that existing PSE methods suffer from a trade-off between speech over-suppression and interference leakage by addressing one problem at the expense of the other. We propose a new PSE model training framework using cross-task knowledge distillation to mitigate this trade-off. Specifically, we utilize a personalized voice activity detector (pVAD) during training to exclude the non-target speech frames that are wrongly identified as containing the target speaker with hard or soft classification. This prevents the PSE model from being too aggressive while still allowing the model to learn to suppress the input speech when it is likely to be spoken by interfering speakers. Comprehensive evaluation results are presented, covering various PSE usage scenarios.