# Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classification

arXiv: https://arxiv.org/abs/2503.11363 · HTML: https://mlsanity.com/paper/2503.11363/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutions](https://mlsanity.com/paper/2005.14623/) (2020)
- [Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation](https://mlsanity.com/paper/2211.04772/) (2022)
- [Low-complexity acoustic scene classification in DCASE 2022 Challenge](https://mlsanity.com/paper/2206.03835/) (2022)
- [Device-Robust Acoustic Scene Classification via Impulse Response Augmentation](https://mlsanity.com/paper/2305.07499/) (2023)
- [QTI Submission to DCASE 2021: residual normalization for device-imbalanced acoustic scene classification with efficient design](https://mlsanity.com/paper/2206.13909/) (2022)
- [Acoustic scene classification using teacher-student learning with soft-labels](https://mlsanity.com/paper/1904.10135/) (2019)

## Built on by (most cited first, 1 of 1)

- [Lightweight and Generalizable Acoustic Scene Representations via Contrastive Fine-Tuning and Distillation](https://mlsanity.com/paper/2510.03728/) (2025)

## Abstract (arXiv)

Knowledge Distillation (KD) is a widespread technique for compressing the knowledge of large models into more compact and efficient models. KD has proved to be highly effective in building well-performing low-complexity Acoustic Scene Classification (ASC) systems and was used in all the top-ranked submissions to this task of the annual DCASE challenge in the past three years. There is extensive research available on establishing the KD process, designing efficient student models, and forming well-performing teacher ensembles. However, less research has been conducted on investigating which teacher model attributes are beneficial for low-complexity students. In this work, we try to close this gap by studying the effects on the student's performance when using different teacher network architectures, varying the teacher model size, training them with different device generalization methods, and applying different ensembling strategies. The results show that teacher model sizes, device generalization methods, the ensembling strategy and the ensemble size are key factors for a well-performing student network.