# Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles

arXiv: https://arxiv.org/abs/2505.22027 · HTML: https://mlsanity.com/paper/2505.22027/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels](https://mlsanity.com/paper/2303.13005/) (2023)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [RespireNet: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting](https://mlsanity.com/paper/2011.00196/) (2020)
- [Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification](https://mlsanity.com/paper/2305.14032/) (2023)
- [Masked Modeling Duo: Towards a Universal Audio Pre-training Framework](https://mlsanity.com/paper/2404.06095/) (2024)
- [Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking](https://mlsanity.com/paper/2406.16148/) (2024)
- [Prototype Learning for Interpretable Respiratory Sound Analysis](https://mlsanity.com/paper/2110.03536/) (2021)
- [Stethoscope-guided Supervised Contrastive Learning for Cross-domain Adaptation on Respiratory Sound Classification](https://mlsanity.com/paper/2312.09603/) (2023)
- [Pretraining Respiratory Sound Representations using Metadata and Contrastive Learning](https://mlsanity.com/paper/2210.16192/) (2022)
- [Lung Sound Classification Using Co-tuning and Stochastic Normalization](https://mlsanity.com/paper/2108.01991/) (2021)
- [Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis](https://mlsanity.com/paper/2203.16141/) (2022)
- [RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory Sound Classification](https://mlsanity.com/paper/2405.02996/) (2024)
- [BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification](https://mlsanity.com/paper/2406.06786/) (2024)
- [Towards Inference Efficient Deep Ensemble Learning](https://mlsanity.com/paper/2301.12378/) (2023)
- [Adversarial Fine-tuning using Generated Respiratory Sound to Address Class Imbalance](https://mlsanity.com/paper/2311.06480/) (2023)

## Built on by (most cited first, 3 of 3)

- [Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classification](https://mlsanity.com/paper/2604.24096/) (2026)
- [Quality Adaptive Angular Margin Learning for Respiratory Sound Classification](https://mlsanity.com/paper/2606.11915/) (2026)
- [Lung-SRAD: Spectral-Aware Regularized Audio DASS with Dual-Axis Patch-Mix Contrastive Learning for Respiratory Sound Classification](https://mlsanity.com/paper/2606.11922/) (2026)

## Abstract (arXiv)

Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.