mlsanityData to 7 Oct 2026

Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

arXiv 2211.04772 ↗First posted 9 Nov 2022cs.SD cs.LG eess.AScited by 25

Not on the map. Why?

Abstract (arXiv)

Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

23 of 29 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017Squeeze-and-Excitation Networkscited by 4,763
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2022A ConvNet for the 2020scited by 3,545
  11. 2019Searching for MobileNetV3cited by 1,984
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
11 more
  1. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  2. 2021AST: Audio Spectrogram Transformercited by 634
  3. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  4. 2022Masked Autoencoders that Listencited by 271
  5. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  6. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  7. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  8. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  9. 2021ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognitioncited by 29
  10. 2021Receptive Field Regularization Techniques for Audio Classification and Tagging with Deep Convolutional Neural Networkscited by 15
  11. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 8; 2026: 5.

25 papers.

Sort:

By area:

  1. 2023Can Language Models Laugh at YouTube Short-form Videos?cited by 11
  2. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretrainingcited by 9
  3. 2025Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillationcited by 6
  4. 2024Generalization in birdsong classification: impact of transfer learning methods and dataset characteristicscited by 5
  5. 2023Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Modelscited by 3
  6. 2024Robust Active Speaker Detection in Noisy Environmentscited by 2
  7. 2024Fusing Audio and Metadata Embeddings Improves Language-based Audio Retrievalcited by 2
  8. 2024Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrievalcited by 2
  9. 2024Spectral and Rhythm Features for Audio Classification with Deep Convolutional Neural Networkscited by 2
  10. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  11. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
  12. 2023Domain Information Control at Inference Time for Acoustic Scene Classificationcited by 1
13 more
  1. 2025Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classificationcited by 1
  2. 2024Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Datacited by 0
  3. 2024Integrating IP Broadcasting with Audio Tags: Workflow and Challengescited by 0
  4. 2024Macformer: Transformer with Random Maclaurin Feature Attentioncited by 0
  5. 2025Exploring Performance-Complexity Trade-Offs in Sound Event Detection Modelscited by 0
  6. 2025TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantizationcited by 0
  7. 2025Comprehensive Evaluation of CNN-Based Audio Tagging Models on Resource-Constrained Devicescited by 0
  8. 2025Compressing Quaternion Convolutional Neural Networks for Audio Classificationcited by 0
  9. 2026The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localizationcited by 0
  10. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Modelscited by 0
  11. 2026Quality Audio Prototyping: a prototype system for unified sound retrieval and procedural generationcited by 0
  12. 2026Detecting Agitation Before Behavioral Escalation in Autistic Youth Through Multimodal Wearable Sensingcited by 0
  13. 2026Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Searchcited by 0