# CED: Consistent ensemble distillation for audio tagging

arXiv: https://arxiv.org/abs/2308.11957 · HTML: https://mlsanity.com/paper/2308.11957/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 24 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 22 of 22)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation](https://mlsanity.com/paper/2211.04772/) (2022)
- [Adapting a ConvNeXt model to audio classification on AudioSet](https://mlsanity.com/paper/2306.00830/) (2023)
- [Dataset balancing can hurt model performance](https://mlsanity.com/paper/2307.00079/) (2023)
- [Unified Keyword Spotting and Audio Tagging on Mobile Devices with Transformers](https://mlsanity.com/paper/2303.01812/) (2023)
- [Pseudo strong labels for large scale weakly supervised audio tagging](https://mlsanity.com/paper/2204.13430/) (2022)
- [Streaming Audio Transformers for Online Audio Tagging](https://mlsanity.com/paper/2305.17834/) (2023)
- [UniKW-AT: Unified Keyword Spotting and Audio Tagging](https://mlsanity.com/paper/2209.11377/) (2022)

## Built on by (most cited first, 29 of 29)

- [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024)
- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning](https://mlsanity.com/paper/2507.02915/) (2025)
- [Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding](https://mlsanity.com/paper/2406.13275/) (2024)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport](https://mlsanity.com/paper/2501.09291/) (2025)
- [Bridging Language Gaps in Audio-Text Retrieval](https://mlsanity.com/paper/2406.07012/) (2024)
- [ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors](https://mlsanity.com/paper/2502.14627/) (2025)
- [MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks](https://mlsanity.com/paper/2507.23511/) (2025)
- [Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models](https://mlsanity.com/paper/2310.15648/) (2023)
- [Improving Anomalous Sound Detection via Low-Rank Adaptation Fine-Tuning of Pre-Trained Audio Models](https://mlsanity.com/paper/2409.07016/) (2024)
- [SupCLAP: Controlling Optimization Trajectory Drift in Audio-Text Contrastive Learning with Support Vector Regularization](https://mlsanity.com/paper/2509.21033/) (2025)
- [FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation](https://mlsanity.com/paper/2507.16696/) (2025)
- [AC/DC: LLM-based Audio Comprehension via Dialogue Continuation](https://mlsanity.com/paper/2506.10312/) (2025)
- [Hashing-Baseline: Rethinking Hashing in the Age of Pretrained Models](https://mlsanity.com/paper/2509.14427/) (2025)
- [LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2601.04658/) (2026)
- [Hierarchical Label Propagation: A Model-Size-Dependent Performance Booster for AudioSet Tagging](https://mlsanity.com/paper/2503.21826/) (2025)
- [Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025](https://mlsanity.com/paper/2506.02088/) (2025)
- [ECHO: Frequency-aware Hierarchical Encoding for Variable-length Signals](https://mlsanity.com/paper/2508.14689/) (2025)
- [ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval](https://mlsanity.com/paper/2512.19703/) (2025)
- [SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training](https://mlsanity.com/paper/2601.12594/) (2026)
- [Sub-Band Spectral Matching with Localized Score Aggregation for Robust Anomalous Sound Detection](https://mlsanity.com/paper/2603.13749/) (2026)
- [EvA: An Evidence-First Audio Understanding Paradigm for LALMs](https://mlsanity.com/paper/2603.27667/) (2026)
- [CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning](https://mlsanity.com/paper/2607.04619/) (2026)
- [Sparse Prototype Code Underlies Classification and Prediction Across Modalities](https://mlsanity.com/paper/2608.15632/) (2026)
- [Finetuning Strategies for Querying Sounds by Vocal Imitation](https://mlsanity.com/paper/2608.19174/) (2026)
- [Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models](https://mlsanity.com/paper/2609.18041/) (2026)
- [REVE: Efficient Hallucination Correction for Large Audio-Language Models via Reused Encoder States](https://mlsanity.com/paper/2609.26028/) (2026)
- [Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding](https://mlsanity.com/paper/2609.28344/) (2026)

## Abstract (arXiv)

Augmentation and knowledge distillation (KD) are well-established techniques employed in audio classification tasks, aimed at enhancing performance and reducing model sizes on the widely recognized Audioset (AS) benchmark. Although both techniques are effective individually, their combined use, called consistent teaching, hasn't been explored before. This paper proposes CED, a simple training framework that distils student models from large teacher ensembles with consistent teaching. To achieve this, CED efficiently stores logits as well as the augmentation methods on disk, making it scalable to large-scale datasets. Central to CED's efficacy is its label-free nature, meaning that only the stored logits are used for the optimization of a student model only requiring 0.3% additional disk space for AS. The study trains various transformer-based models, including a 10M parameter model achieving a 49.0 mean average precision (mAP) on AS. Pretrained models and code are available at https://github.com/RicherMans/CED.