# Joint Feature and Output Distillation for Low-complexity Acoustic Scene Classification

arXiv: https://arxiv.org/abs/2507.19557 · HTML: https://mlsanity.com/paper/2507.19557/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 25 Jul 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 8 of 8)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023)
- [Receptive-field-regularized CNN variants for acoustic scene classification](https://mlsanity.com/paper/1909.02859/) (2019)
- [Data Efficient Acoustic Scene Classification using Teacher-Informed Confusing Class Instruction](https://mlsanity.com/paper/2409.11964/) (2024)

## Abstract (arXiv)

This report presents a dual-level knowledge distillation framework with multi-teacher guidance for low-complexity acoustic scene classification (ASC) in DCASE2025 Task 1. We propose a distillation strategy that jointly transfers both soft logits and intermediate feature representations. Specifically, we pre-trained PaSST and CP-ResNet models as teacher models. Logits from teachers are averaged to generate soft targets, while one CP-ResNet is selected for feature-level distillation. This enables the compact student model (CP-Mobile) to capture both semantic distribution and structural information from teacher guidance. Experiments on the TAU Urban Acoustic Scenes 2022 Mobile dataset (development set) demonstrate that our submitted systems achieve up to 59.30% accuracy.