# ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression

arXiv: https://arxiv.org/abs/2305.16665 · HTML: https://mlsanity.com/paper/2305.16665/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 26 May 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matching](https://mlsanity.com/paper/2102.02973/) (2021)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [Self-supervised Knowledge Distillation Using Singular Value Decomposition](https://mlsanity.com/paper/1807.06819/) (2018)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [Conditional Teacher-Student Learning](https://mlsanity.com/paper/1904.12399/) (2019)
- [Towards efficient models for real-time deep noise suppression](https://mlsanity.com/paper/2101.09249/) (2021)
- [RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation](https://mlsanity.com/paper/2109.10164/) (2021)
- [Dual-Signal Transformation LSTM Network for Real-Time Noise Suppression](https://mlsanity.com/paper/2005.07551/) (2020)
- [Interactive Speech and Noise Modeling for Speech Enhancement](https://mlsanity.com/paper/2012.09408/) (2020)
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022)
- [Sub-Band Knowledge Distillation Framework for Speech Enhancement](https://mlsanity.com/paper/2005.14435/) (2020)
- [Monaural Speech Enhancement with Complex Convolutional Block Attention Module and Joint Time Frequency Losses](https://mlsanity.com/paper/2102.01993/) (2021)
- [Real-time Monaural Speech Enhancement With Short-time Discrete Cosine Transform](https://mlsanity.com/paper/2102.04629/) (2021)
- [SNR-Based Teachers-Student Technique for Speech Enhancement](https://mlsanity.com/paper/2005.14441/) (2020)

## Built on by (most cited first, 4 of 4)

- [Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement](https://mlsanity.com/paper/2408.04267/) (2024)
- [Dynamic Frequency-Adaptive Knowledge Distillation for Speech Enhancement](https://mlsanity.com/paper/2502.04711/) (2025)
- [Leveraging Local and Global Knowledge Integration with Time-Frequency Calibrated Distillation for Speech Enhancement](https://mlsanity.com/paper/2506.13127/) (2025)
- [DISPATCH: Distilling Selective Patches for Speech Enhancement](https://mlsanity.com/paper/2509.15922/) (2025)

## Abstract (arXiv)

Noise suppression (NS) models have been widely applied to enhance speech quality. Recently, Deep Learning-Based NS, which we denote as Deep Noise Suppression (DNS), became the mainstream NS method due to its excelling performance over traditional ones. However, DNS models face 2 major challenges for supporting the real-world applications. First, high-performing DNS models are usually large in size, causing deployment difficulties. Second, DNS models require extensive training data, including noisy audios as inputs and clean audios as labels. It is often difficult to obtain clean labels for training DNS models. We propose the use of knowledge distillation (KD) to resolve both challenges. Our study serves 2 main purposes. To begin with, we are among the first to comprehensively investigate mainstream KD techniques on DNS models to resolve the two challenges. Furthermore, we propose a novel Attention-Based-Compression KD method that outperforms all investigated mainstream KD frameworks on DNS task.