# Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation

arXiv: https://arxiv.org/abs/2105.03544 · HTML: https://mlsanity.com/paper/2105.03544/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 8 May 2021.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Asteroid: the PyTorch-based audio source separation toolkit for researchers](https://mlsanity.com/paper/2005.04132/) (2020)
- [Large-Scale Domain Adaptation via Teacher-Student Learning](https://mlsanity.com/paper/1708.05466/) (2017)
- [Unsupervised training of a deep clustering model for multichannel blind source separation](https://mlsanity.com/paper/1904.01340/) (2019)
- [Towards Zero-shot Learning for Automatic Phonemic Transcription](https://mlsanity.com/paper/2002.11781/) (2020)
- [Zero-shot Learning for Audio-based Music Classification and Tagging](https://mlsanity.com/paper/1907.02670/) (2019)
- [Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information](https://mlsanity.com/paper/1811.01531/) (2018)
- [Sparse Mixture of Local Experts for Efficient Speech Enhancement](https://mlsanity.com/paper/2005.08128/) (2020)
- [Zero-Shot Learning for Semantic Utterance Classification](https://mlsanity.com/paper/1401.0509/) (2013)
- [Speech Enhancement using a Deep Mixture of Experts](https://mlsanity.com/paper/1703.09302/) (2017)

## Built on by (most cited first, 10 of 10)

- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022)
- [Continual self-training with bootstrapped remixing for speech enhancement](https://mlsanity.com/paper/2110.10103/) (2021)
- [BLOOM-Net: Blockwise Optimization for Masking Networks Toward Scalable and Efficient Speech Enhancement](https://mlsanity.com/paper/2111.09372/) (2021)
- [Test-Time Training for Speech Enhancement](https://mlsanity.com/paper/2508.01847/) (2025)
- [Augmented Contrastive Clustering with Uncertainty-Aware Prototyping for Time Series Test Time Adaptation](https://mlsanity.com/paper/2501.01472/) (2025)
- [An Investigation of Test-time Adaptation for Audio Classification under Background Noise](https://mlsanity.com/paper/2507.15523/) (2025)
- [DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformers](https://mlsanity.com/paper/2603.21608/) (2026)
- [Knowledge Distillation for Efficient Acoustic Echo Control](https://mlsanity.com/paper/2608.25596/) (2026)
- [Test-time adaptation for speech enhancement with an autoregressive speech prior](https://mlsanity.com/paper/2609.03622/) (2026)

## Abstract (arXiv)

In realistic speech enhancement settings for end-user devices, we often encounter only a few speakers and noise types that tend to reoccur in the specific acoustic environment. We propose a novel personalized speech enhancement method to adapt a compact denoising model to the test-time specificity. Our goal in this test-time adaptation is to utilize no clean speech target of the test speaker, thus fulfilling the requirement for zero-shot learning. To complement the lack of clean utterance, we employ the knowledge distillation framework. Instead of the missing clean utterance target, we distill the more advanced denoising results from an overly large teacher model, and use it as the pseudo target to train the small student model. This zero-shot learning procedure circumvents the process of collecting users' clean speech, a process that users are reluctant to comply due to privacy concerns and technical difficulty of recording clean voice. Experiments on various test-time conditions show that the proposed personalization method achieves significant performance gains compared to larger baseline networks trained from a large speaker- and noise-agnostic datasets. In addition, since the compact personalized models can outperform larger general-purpose models, we claim that the proposed method performs model compression with no loss of denoising performance.