# Comparison of Knowledge Distillation Methods for Low-complexity Multi-microphone Speech Enhancement using the FT-JNF Architecture

arXiv: https://arxiv.org/abs/2507.19208 · HTML: https://mlsanity.com/paper/2507.19208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation](https://mlsanity.com/paper/2211.12433/) (2022)
- [Deep Filtering: Signal Extraction and Reconstruction Using Complex Time-Frequency Filters](https://mlsanity.com/paper/1904.08369/) (2019)
- [STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency](https://mlsanity.com/paper/2204.09911/) (2022)
- [Insights Into Deep Non-linear Filters for Improved Multi-channel Speech Enhancement](https://mlsanity.com/paper/2206.13310/) (2022)
- [Uformer: A Unet based dilated complex & real dual-path conformer network for simultaneous speech enhancement and dereverberation](https://mlsanity.com/paper/2111.06015/) (2021)
- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023)
- [Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancement](https://mlsanity.com/paper/2406.04269/) (2024)
- [Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement](https://mlsanity.com/paper/2408.04267/) (2024)
- [Microphone Array Signal Processing and Deep Learning for Speech Enhancement](https://mlsanity.com/paper/2501.07215/) (2025)

## Abstract (arXiv)

Multi-microphone speech enhancement using deep neural networks (DNNs) has significantly progressed in recent years. However, many proposed DNN-based speech enhancement algorithms cannot be implemented on devices with limited hardware resources. Only lowering the complexity of such systems by reducing the number of parameters often results in worse performance. Knowledge Distillation (KD) is a promising approach for reducing DNN model size while preserving performance. In this paper, we consider the recently proposed Frequency-Time Joint Non-linear Filter (FT-JNF) architecture and investigate several KD methods to train smaller (student) models from a large pre-trained (teacher) model. Five KD methods are evaluated using direct output matching, the self-similarity of intermediate layers, and fused multi-layer losses. Experimental results on a simulated dataset using a compact array with five microphones show that three KD methods substantially improve the performance of student models compared to training without KD. A student model with only 25% of the teacher model's parameters achieves comparable PESQ scores at 0 dB SNR. Furthermore, a reduction of up to 96% in model size can be achieved with only a minimal decrease in PESQ scores.