# Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation

arXiv: https://arxiv.org/abs/2204.00771 · HTML: https://mlsanity.com/paper/2204.00771/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 2 Apr 2022.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 19 of 19)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Shrinking Bigfoot: Reducing wav2vec 2.0 footprint](https://mlsanity.com/paper/2103.15760/) (2021)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [Can Model Compression Improve NLP Fairness](https://mlsanity.com/paper/2201.08542/) (2022)
- [Personalized PercepNet: Real-time, Low-complexity Target Voice Separation and Enhancement](https://mlsanity.com/paper/2106.04129/) (2021)
- [Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation](https://mlsanity.com/paper/2105.03544/) (2021)
- [Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement](https://mlsanity.com/paper/2106.02896/) (2021)
- [One model to enhance them all: array geometry agnostic multi-channel personalized speech enhancement](https://mlsanity.com/paper/2110.10330/) (2021)
- [SNR-Based Teachers-Student Technique for Speech Enhancement](https://mlsanity.com/paper/2005.14441/) (2020)

## Built on by (most cited first, 12 of 12)

- [Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inference](https://mlsanity.com/paper/2312.03464/) (2023)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility](https://mlsanity.com/paper/2409.09357/) (2024)
- [Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation](https://mlsanity.com/paper/2210.12635/) (2022)
- [ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression](https://mlsanity.com/paper/2305.16665/) (2023)
- [RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2407.07825/) (2024)
- [Dynamic Frequency-Adaptive Knowledge Distillation for Speech Enhancement](https://mlsanity.com/paper/2502.04711/) (2025)
- [Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment](https://mlsanity.com/paper/2501.18157/) (2025)
- [Breaking the trade-off in personalized speech enhancement with cross-task knowledge distillation](https://mlsanity.com/paper/2211.02944/) (2022)
- [Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues](https://mlsanity.com/paper/2311.14275/) (2023)
- [Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance](https://mlsanity.com/paper/2505.03442/) (2025)
- [Contrastive Knowledge Distillation for Embedding Refinement in Personalized Speech Enhancement](https://mlsanity.com/paper/2601.16235/) (2026)

## Abstract (arXiv)

This paper investigates how to improve the runtime speed of personalized speech enhancement (PSE) networks while maintaining the model quality. Our approach includes two aspects: architecture and knowledge distillation (KD). We propose an end-to-end enhancement (E3Net) model architecture, which is 3× faster than a baseline STFT-based model. Besides, we use KD techniques to develop compressed student models without significantly degrading quality. In addition, we investigate using noisy data without reference clean signals for training the student models, where we combine KD with multi-task learning (MTL) using automatic speech recognition (ASR) loss. Our results show that E3Net provides better speech and transcription quality with a lower target speaker over-suppression (TSOS) rate than the baseline model. Furthermore, we show that the KD methods can yield student models that are 2-4× faster than the teacher and provides reasonable quality. Combining KD and MTL improves the ASR and TSOS metrics without degrading the speech quality.