mlsanityData to 14 Oct 2026

Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles

arXiv 2505.22027 ↗First posted 28 May 2025cs.SD cs.AI eess.AScited by 3

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 25 of its 31 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

25 of 31 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  7. 2021AST: Audio Spectrogram Transformercited by 643
  8. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  9. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  10. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  11. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  12. 2025Distillation Scaling Lawscited by 67
13 more
  1. 2020RespireNet: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Settingcited by 26
  2. 2023Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classificationcited by 24
  3. 2024Masked Modeling Duo: Towards a Universal Audio Pre-training Frameworkcited by 23
  4. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  5. 2021Prototype Learning for Interpretable Respiratory Sound Analysiscited by 21
  6. 2023Stethoscope-guided Supervised Contrastive Learning for Cross-domain Adaptation on Respiratory Sound Classificationcited by 16
  7. 2022Pretraining Respiratory Sound Representations using Metadata and Contrastive Learningcited by 15
  8. 2021Lung Sound Classification Using Co-tuning and Stochastic Normalizationcited by 14
  9. 2022Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysiscited by 13
  10. 2024RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory Sound Classificationcited by 12
  11. 2024BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classificationcited by 12
  12. 2023Towards Inference Efficient Deep Ensemble Learningcited by 11
  13. 2023Adversarial Fine-tuning using Generated Respiratory Sound to Address Class Imbalancecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classificationcited by 0
  2. 2026Quality Adaptive Angular Margin Learning for Respiratory Sound Classificationcited by 0
  3. 2026Lung-SRAD: Spectral-Aware Regularized Audio DASS with Dual-Axis Patch-Mix Contrastive Learning for Respiratory Sound Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification
  2. 2026Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment
  3. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction
  4. 2020CNN-MoE based framework for classification of respiratory anomalies and lung disease detection ↗
  5. 2025Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classification
  6. 2021Robust Deep Learning Frameworks for Acoustic Scene and Respiratory Sound Classification
  7. 2026Synthetic Data Augmentation for Medical Audio Classification: A Preliminary Evaluation
  8. 2026Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis