# Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement

arXiv: https://arxiv.org/abs/2408.04267 · HTML: https://mlsanity.com/paper/2408.04267/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 28 of 28)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network](https://mlsanity.com/paper/1911.04697/) (2019)
- [DCCRN+: Channel-wise Subband DCCRN with SNR Estimation for Speech Enhancement](https://mlsanity.com/paper/2106.08672/) (2021)
- [S-DCCRN: Super Wide Band DCCRN with learnable complex feature for speech enhancement](https://mlsanity.com/paper/2111.08387/) (2021)
- [Uformer: A Unet based dilated complex & real dual-path conformer network for simultaneous speech enhancement and dereverberation](https://mlsanity.com/paper/2111.06015/) (2021)
- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023)
- [DESNet: A Multi-channel Network for Simultaneous Speech Dereverberation, Enhancement and Separation](https://mlsanity.com/paper/2011.02131/) (2020)
- [Multi-View Attention Transfer for Efficient Speech Enhancement](https://mlsanity.com/paper/2208.10367/) (2022)
- [Online Monaural Speech Enhancement Using Delayed Subband LSTM](https://mlsanity.com/paper/2005.05037/) (2020)
- [DCCRN-KWS: an audio bias based model for noise robust small-footprint keyword spotting](https://mlsanity.com/paper/2305.12331/) (2023)
- [ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression](https://mlsanity.com/paper/2305.16665/) (2023)
- [A Deep Representation Learning-based Speech Enhancement Method Using Complex Convolution Recurrent Variational Autoencoder](https://mlsanity.com/paper/2312.09620/) (2023)
- [Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge](https://mlsanity.com/paper/2303.07621/) (2023)
- [CheapNET: Improving Light-weight speech enhancement network by projected loss function](https://mlsanity.com/paper/2311.15959/) (2023)
- [Causal Signal-Based DCCRN with Overlapped-Frame Prediction for Online Speech Enhancement](https://mlsanity.com/paper/2309.03684/) (2023)

## Built on by (most cited first, 5 of 5)

- [Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance](https://mlsanity.com/paper/2505.03442/) (2025)
- [Leveraging Local and Global Knowledge Integration with Time-Frequency Calibrated Distillation for Speech Enhancement](https://mlsanity.com/paper/2506.13127/) (2025)
- [Comparison of Knowledge Distillation Methods for Low-complexity Multi-microphone Speech Enhancement using the FT-JNF Architecture](https://mlsanity.com/paper/2507.19208/) (2025)
- [CabinSep: IR-Augmented Mask-Based MVDR for Real-Time In-Car Speech Separation with Distributed Heterogeneous Arrays](https://mlsanity.com/paper/2509.01399/) (2025)
- [Batch Normalization-Free Fully Integer Quantized Neural Networks via Progressive Tandem Learning](https://mlsanity.com/paper/2512.16476/) (2025)

## Abstract (arXiv)

The deep complex convolution recurrent network (DCCRN) achieves excellent speech enhancement performance by utilizing the audio spectrum's complex features. However, it has a large number of model parameters. We propose a smaller model, Distil-DCCRN, which has only 30% of the parameters compared to the DCCRN. To ensure that the performance of Distil-DCCRN matches that of the DCCRN, we employ the knowledge distillation (KD) method to use a larger teacher model to help train a smaller student model. We design a knowledge distillation (KD) method, integrating attention transfer and Kullback-Leibler divergence (AT-KL) to train the student model Distil-DCCRN. Additionally, we use a model with better performance and a more complicated structure, Uformer, as the teacher model. Unlike previous KD approaches that mainly focus on model outputs, our method also leverages the intermediate features from the models' middle layers, facilitating rich knowledge transfer across different structured models despite variations in layer configurations and discrepancies in the channel and time dimensions of intermediate features. Employing our AT-KL approach, Distil-DCCRN outperforms DCCRN as well as several other competitive models in both PESQ and SI-SNR metrics on the DNS test set and achieves comparable results to DCCRN in DNSMOS.