# DISPATCH: Distilling Selective Patches for Speech Enhancement

arXiv: https://arxiv.org/abs/2509.15922 · HTML: https://mlsanity.com/paper/2509.15922/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [A Comprehensive Survey on Knowledge Distillation](https://mlsanity.com/paper/2503.12067/) (2025)
- [Uncertainty-Aware Multi-Shot Knowledge Distillation for Image-Based Object Re-Identification](https://mlsanity.com/paper/2001.05197/) (2020)
- [An empirical study of Conv-TasNet](https://mlsanity.com/paper/2002.08688/) (2020)
- [Teacher-Student MixIT for Unsupervised and Semi-supervised Speech Separation](https://mlsanity.com/paper/2106.07843/) (2021)
- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023)
- [Sub-Band Knowledge Distillation Framework for Speech Enhancement](https://mlsanity.com/paper/2005.14435/) (2020)
- [DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement](https://mlsanity.com/paper/2212.07570/) (2022)
- [ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression](https://mlsanity.com/paper/2305.16665/) (2023)
- [Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification](https://mlsanity.com/paper/2309.14838/) (2023)
- [Maximizing Discrimination Capability of Knowledge Distillation with Energy Function](https://mlsanity.com/paper/2311.14334/) (2023)
- [Dynamic Frequency-Adaptive Knowledge Distillation for Speech Enhancement](https://mlsanity.com/paper/2502.04711/) (2025)
- [Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems](https://mlsanity.com/paper/2111.03842/) (2021)

## Abstract (arXiv)

In speech enhancement, knowledge distillation (KD) compresses models by transferring a high-capacity teacher's knowledge to a compact student. However, conventional KD methods train the student to mimic the teacher's output entirely, which forces the student to imitate the regions where the teacher performs poorly and to apply distillation to the regions where the student already performs well, which yields only marginal gains. We propose Distilling Selective Patches (DISPatch), a KD framework for speech enhancement that applies the distillation loss to spectrogram patches where the teacher outperforms the student, as determined by a Knowledge Gap Score. This approach guides optimization toward areas with the most significant potential for student improvement while minimizing the influence of regions where the teacher may provide unreliable instruction. Furthermore, we introduce Multi-Scale Selective Patches (MSSP), a frequency-dependent method that uses different patch sizes across low- and high-frequency bands to account for spectral heterogeneity. We incorporate DISPatch into conventional KD methods and observe consistent gains in compact students. Moreover, integrating DISPatch and MSSP into a state-of-the-art frequency-dependent KD method considerably improves performance across all metrics.