mlsanityData to 7 Oct 2026

CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification

arXiv 2203.06760 ↗First posted 13 Mar 2022cs.SD cs.AI eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Audio classification is an active research area with a wide range of applications. Over the past decade, convolutional neural networks (CNNs) have been the de-facto standard building block for end-to-end audio classification models. Recently, neural networks based solely on self-attention mechanisms such as the Audio Spectrogram Transformer (AST) have been shown to outperform CNNs. In this paper, we find an intriguing interaction between the two very different models - CNN and AST models are good teachers for each other. When we use either of them as the teacher and train the other model as the student via knowledge distillation (KD), the performance of the student model noticeably improves, and in many cases, is better than the teacher model. In our experiments with this CNN/Transformer Cross-Model Knowledge Distillation (CMKD) method we achieve new state-of-the-art performance on FSD50K, AudioSet, and ESC-50.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 52 of its 85 references that are in the map. First posted on arXiv 13 Mar 2022.

Builds on

52 of 85 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2022A ConvNet for the 2020scited by 3,545
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  6. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  7. 2021AST: Audio Spectrogram Transformercited by 634
  8. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  10. 2021Intriguing Properties of Vision Transformerscited by 446
  11. 2021Early Convolutions Help Transformers See Bettercited by 395
  12. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  13. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  14. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  15. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  16. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  17. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  18. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  19. 2022Patches Are All You Need?cited by 201
  20. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  21. 2016Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classificationcited by 84
  22. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  23. 2020Rethinking CNN Models for Audio Classificationcited by 48
  24. 2020Transferring Inductive Biases through Knowledge Distillationcited by 41
  25. 2021Audio Transformerscited by 41
  26. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  27. 2021ConTNet: Why not use convolution and transformer at the same time?cited by 38
  28. 2018A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labelingcited by 37
  29. 2020ESResNet: Environmental Sound Classification Based on Visual Domain Modelscited by 31
  30. 2019Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimizationcited by 29
  31. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  32. 2017Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networkscited by 14
  33. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 13
  34. 2018A Closer Look at Weak Label Learning for Audio Eventscited by 11
  35. 2019Scalable Syntax-Aware Language Models Using Knowledge Distillationcited by 8
  36. 2019Urban Sound Tagging using Convolutional Neural Networkscited by 7
  37. 2020Addressing Missing Labels in Large-Scale Sound Event Recognition Using a Teacher-Student Framework With Loss Maskingcited by 7
  38. 2021Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networkscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

12 papers.

  1. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 25
  2. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  3. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  4. 2024Generalization in birdsong classification: impact of transfer learning methods and dataset characteristicscited by 5
  5. 2024FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillationcited by 4
  6. 2025GATE3D: Generalized Attention-based Task-synergized Estimation in 3D*cited by 4
  7. 2022C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrievalcited by 3
  8. 2023Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Modelscited by 3
  9. 2023Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervisioncited by 2
  10. 2023Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learningcited by 0
  11. 2024Revisiting Monocular 3D Object Detection with Depth Thickness Fieldcited by 0
  12. 2026RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detectioncited by 0