# Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

arXiv: https://arxiv.org/abs/2211.04772 · HTML: https://mlsanity.com/paper/2211.04772/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/2106.01621/) (2021)
- [Receptive Field Regularization Techniques for Audio Classification and Tagging with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2105.12395/) (2021)
- [CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification](https://mlsanity.com/paper/2203.06760/) (2022)

## Built on by (most cited first, 27 of 27)

- [CED: Consistent ensemble distillation for audio tagging](https://mlsanity.com/paper/2308.11957/) (2023)
- [Can Language Models Laugh at YouTube Short-form Videos?](https://mlsanity.com/paper/2310.14159/) (2023)
- [TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining](https://mlsanity.com/paper/2505.07609/) (2025)
- [Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillation](https://mlsanity.com/paper/2505.14705/) (2025)
- [Generalization in birdsong classification: impact of transfer learning methods and dataset characteristics](https://mlsanity.com/paper/2409.15383/) (2024)
- [Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models](https://mlsanity.com/paper/2310.15648/) (2023)
- [Robust Active Speaker Detection in Noisy Environments](https://mlsanity.com/paper/2403.19002/) (2024)
- [Fusing Audio and Metadata Embeddings Improves Language-based Audio Retrieval](https://mlsanity.com/paper/2406.15897/) (2024)
- [Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval](https://mlsanity.com/paper/2408.11641/) (2024)
- [Spectral and Rhythm Features for Audio Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2410.06927/) (2024)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.21556/) (2025)
- [Domain Information Control at Inference Time for Acoustic Scene Classification](https://mlsanity.com/paper/2306.08010/) (2023)
- [Efficient Audio Captioning with Encoder-Level Knowledge Distillation](https://mlsanity.com/paper/2407.14329/) (2024)
- [Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classification](https://mlsanity.com/paper/2503.11363/) (2025)
- [Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Data](https://mlsanity.com/paper/2402.10100/) (2024)
- [Integrating IP Broadcasting with Audio Tags: Workflow and Challenges](https://mlsanity.com/paper/2407.15423/) (2024)
- [Macformer: Transformer with Random Maclaurin Feature Attention](https://mlsanity.com/paper/2408.11656/) (2024)
- [Exploring Performance-Complexity Trade-Offs in Sound Event Detection Models](https://mlsanity.com/paper/2503.11373/) (2025)
- [TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization](https://mlsanity.com/paper/2509.00914/) (2025)
- [Comprehensive Evaluation of CNN-Based Audio Tagging Models on Resource-Constrained Devices](https://mlsanity.com/paper/2509.14049/) (2025)
- [Compressing Quaternion Convolutional Neural Networks for Audio Classification](https://mlsanity.com/paper/2510.21388/) (2025)
- [The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization](https://mlsanity.com/paper/2601.03227/) (2026)
- [S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models](https://mlsanity.com/paper/2604.24933/) (2026)
- [Quality Audio Prototyping: a prototype system for unified sound retrieval and procedural generation](https://mlsanity.com/paper/2606.00629/) (2026)
- [Detecting Agitation Before Behavioral Escalation in Autistic Youth Through Multimodal Wearable Sensing](https://mlsanity.com/paper/2609.24791/) (2026)
- [Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search](https://mlsanity.com/paper/2609.31024/) (2026)

## Abstract (arXiv)

Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT