mlsanityData to 14 Oct 2026

CED: Consistent ensemble distillation for audio tagging

arXiv 2308.11957 ↗First posted 23 Aug 2023cs.SD eess.AScited by 29

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Augmentation and knowledge distillation (KD) are well-established techniques employed in audio classification tasks, aimed at enhancing performance and reducing model sizes on the widely recognized Audioset (AS) benchmark. Although both techniques are effective individually, their combined use, called consistent teaching, hasn't been explored before. This paper proposes CED, a simple training framework that distils student models from large teacher ensembles with consistent teaching. To achieve this, CED efficiently stores logits as well as the augmentation methods on disk, making it scalable to large-scale datasets. Central to CED's efficacy is its label-free nature, meaning that only the stored logits are used for the optimization of a student model only requiring 0.3% additional disk space for AS. The study trains various transformer-based models, including a 10M parameter model achieving a 49.0 mean average precision (mAP) on AS. Pretrained models and code are available at https://github.com/RicherMans/CED.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 22 of its 24 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

22 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation9 Nov

Other references · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2021AST: Audio Spectrogram Transformercited by 643
  6. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  7. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  8. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  9. 2022Masked Autoencoders that Listencited by 273
  10. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  11. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 234
  12. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
9 more
  1. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  2. 2022MAViL: Masked Audio-Video Learnerscited by 59
  3. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 51
  4. 2023Adapting a ConvNeXt model to audio classification on AudioSetcited by 14
  5. 2023Dataset balancing can hurt model performancecited by 5
  6. 2023Unified Keyword Spotting and Audio Tagging on Mobile Devices with Transformerscited by 4
  7. 2022Pseudo strong labels for large scale weakly supervised audio taggingcited by 3
  8. 2023Streaming Audio Transformers for Online Audio Taggingcited by 2
  9. 2022UniKW-AT: Unified Keyword Spotting and Audio Taggingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 13; 2026: 10.

29 papers.

Sort:

By area:

  1. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  2. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 37
  3. 2025Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learningcited by 17
  4. 2024Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encodingcited by 14
  5. 2025GLAP: General contrastive audio-text pretraining across domains and languagescited by 10
  6. 2025LAVCap: LLM-based Audio-Visual Captioning using Optimal Transportcited by 7
  7. 2024Bridging Language Gaps in Audio-Text Retrievalcited by 6
  8. 2025ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errorscited by 6
  9. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  10. 2023Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Modelscited by 3
  11. 2024Improving Anomalous Sound Detection via Low-Rank Adaptation Fine-Tuning of Pre-Trained Audio Modelscited by 3
  12. 2025SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularizationcited by 3
17 more
  1. 2025FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representationcited by 2
  2. 2025AC/DC: LLM-based Audio Comprehension via Dialogue Continuationcited by 1
  3. 2025Hashing-Baseline: Rethinking Hashing in the Age of Pretrained Modelscited by 1
  4. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergencecited by 1
  5. 2025Hierarchical Label Propagation: A Model-Size-Dependent Performance Booster for AudioSet Taggingcited by 0
  6. 2025Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025cited by 0
  7. 2025ECHO: Frequency-aware Hierarchical Encoding for Variable-length Signalscited by 0
  8. 2025ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrievalcited by 0
  9. 2026SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Trainingcited by 0
  10. 2026Sub-Band Spectral Matching with Localized Score Aggregation for Robust Anomalous Sound Detectioncited by 0
  11. 2026EvA: An Evidence-First Audio Understanding Paradigm for LALMscited by 0
  12. 2026CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioningcited by 0
  13. 2026Sparse Prototype Code Underlies Classification and Prediction Across Modalitiescited by 0
  14. 2026Finetuning Strategies for Querying Sounds by Vocal Imitationcited by 0
  15. 2026Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Modelscited by 0
  16. 2026REVE: Efficient Hallucination Correction for Large Audio-Language Models via Reused Encoder Statescited by 0
  17. 2026Mizar: A 159M-Parameter Audio-Language Model for Audio Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classification
  2. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification
  3. 2024Integrated Multi-Level Knowledge Distillation for Enhanced Speaker Verification
  4. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
  5. 2025Distilling a speech and music encoder with task arithmetic
  6. 2022Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation
  7. 2026Learning from imperfect teachers for low-resource acoustic generalization
  8. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition